From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.0 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24B1EC43387 for ; Thu, 20 Dec 2018 09:28:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E57F72177E for ; Thu, 20 Dec 2018 09:28:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1545298109; bh=P6Qw+JDMdiZsKDYcMdSk/EEj7gysmgVD7w5ptHCjeuM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=hG1L+2UBcjW3PpAQfTT+zWsIi1RLIgVc+Yuk1rR+SB9BQR05pAhiCaGb1GvvNvb3a np4w7maN6iwsSgyk0nKsE3ShBI6hIA/CbcKoG8bXOEtCNLDnnJsK7Y2jmB/gKc7NtE xIHlWFfPoNvJlCe806+iBom7E42COOanPiloZstI= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732512AbeLTJ21 (ORCPT ); Thu, 20 Dec 2018 04:28:27 -0500 Received: from mail.kernel.org ([198.145.29.99]:46160 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730044AbeLTJ2Z (ORCPT ); Thu, 20 Dec 2018 04:28:25 -0500 Received: from localhost (5356596B.cm-6-7b.dynamic.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E577720989; Thu, 20 Dec 2018 09:28:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1545298104; bh=P6Qw+JDMdiZsKDYcMdSk/EEj7gysmgVD7w5ptHCjeuM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gZEnmTQUt0asjGT6AJ1NDlsczsdlCtuD/LAigKIx12ETyJF3XaDtDA4AUs6Dy33zv QPLJ/jftCLFGoLkn6BUvCyp9yzJ2+Gi7HdeCT8wC9yBfMI2mIbGjYzPQcIdpWr7iu6 IJn/Y/4ZDGHjJg0buL2DcvJGr0QO+JrQoEf7ic1Y= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Will Deacon , "Peter Zijlstra (Intel)" , Waiman Long , Linus Torvalds , Thomas Gleixner , boqun.feng@gmail.com, linux-arm-kernel@lists.infradead.org, paulmck@linux.vnet.ibm.com, Ingo Molnar , Sebastian Andrzej Siewior , Sasha Levin Subject: [PATCH 4.14 26/72] locking/qspinlock: Kill cmpxchg() loop when claiming lock from head of queue Date: Thu, 20 Dec 2018 10:18:25 +0100 Message-Id: <20181220085923.364056708@linuxfoundation.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20181220085922.332225035@linuxfoundation.org> References: <20181220085922.332225035@linuxfoundation.org> User-Agent: quilt/0.65 X-stable: review X-Patchwork-Hint: ignore MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 4.14-stable review patch. If anyone has any objections, please let me know. ------------------ commit c61da58d8a9ba9238250a548f00826eaf44af0f7 upstream. When a queued locker reaches the head of the queue, it claims the lock by setting _Q_LOCKED_VAL in the lockword. If there isn't contention, it must also clear the tail as part of this operation so that subsequent lockers can avoid taking the slowpath altogether. Currently this is expressed as a cmpxchg() loop that practically only runs up to two iterations. This is confusing to the reader and unhelpful to the compiler. Rewrite the cmpxchg() loop without the loop, so that a failed cmpxchg() implies that there is contention and we just need to write to _Q_LOCKED_VAL without considering the rest of the lockword. Signed-off-by: Will Deacon Acked-by: Peter Zijlstra (Intel) Acked-by: Waiman Long Cc: Linus Torvalds Cc: Thomas Gleixner Cc: boqun.feng@gmail.com Cc: linux-arm-kernel@lists.infradead.org Cc: paulmck@linux.vnet.ibm.com Link: http://lkml.kernel.org/r/1524738868-31318-7-git-send-email-will.deacon@arm.com Signed-off-by: Ingo Molnar Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Sasha Levin --- kernel/locking/qspinlock.c | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index 7bd053e528c2..841550dfb7b8 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -465,24 +465,21 @@ locked: * and nobody is pending, clear the tail code and grab the lock. * Otherwise, we only need to grab the lock. */ - for (;;) { - /* In the PV case we might already have _Q_LOCKED_VAL set */ - if ((val & _Q_TAIL_MASK) != tail || (val & _Q_PENDING_MASK)) { - set_locked(lock); - break; - } + + /* In the PV case we might already have _Q_LOCKED_VAL set */ + if ((val & _Q_TAIL_MASK) == tail) { /* * The smp_cond_load_acquire() call above has provided the - * necessary acquire semantics required for locking. At most - * two iterations of this loop may be ran. + * necessary acquire semantics required for locking. */ old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL); if (old == val) - goto release; /* No contention */ - - val = old; + goto release; /* No contention */ } + /* Either somebody is queued behind us or _Q_PENDING_VAL is set */ + set_locked(lock); + /* * contended path; wait for next if not observed yet, release. */ -- 2.19.1