From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.7 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CADFC43331 for ; Fri, 6 Sep 2019 14:37:20 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0F33D206A1 for ; Fri, 6 Sep 2019 14:37:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="VJHvLT6a"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="NZhrmRf1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0F33D206A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Yaa2unJDAMbEgPw3yEMcJpnL+R++fJvZ4oj46mRjjvs=; b=VJHvLT6afNS/ZI Ok9F5B9Phh8Iqmsf/Amlor7HiZr4FGeJCsY2rO600fbdw+aF2LPagVy4XdOSD52Lpam46kmMYTM5f Z5vu7lIeV+B3vVGG5pEAZ08XraduZgSiiDLbBgLw42b13fd6SchUjHDM1mg77yNsV8+O1dU8qLYIk PHJCoZ4bHQzTaJ1j6g0T7OjlS7oRa2Glmb6X/YcZxW6Wj8+QHk+rACQan4TFfKoSUn7Rvr/ewxhBP yaXDSR1orkUlSt9eAiYtku7+q4NyxXyq+UbVm5+1Yu2Y+mCsId8RQldVFHj7bNwyiZ1iYvZmoMYpk ZT48WxmalF1TFTpj1zdQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1i6FMT-0001PN-DK; Fri, 06 Sep 2019 14:37:17 +0000 Received: from aserp2120.oracle.com ([141.146.126.78]) by bombadil.infradead.org with esmtps (Exim 4.92 #3 (Red Hat Linux)) id 1i6FMQ-0001P1-Gm for linux-arm-kernel@lists.infradead.org; Fri, 06 Sep 2019 14:37:15 +0000 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x86EY7Be032545; Fri, 6 Sep 2019 14:36:45 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=corp-2019-08-05; bh=w5Vu/fbczNwoQcE3H6hN/D9dx7OzQsmQPWGC5Uj8C14=; b=NZhrmRf1W50Q/ZfNJSn3dGFcD7saeYNzsPB0KvZ9at5iuo/aGmDpqd1yx1LPDLf2tm4t bO2LdLi9AmxA5zRZrFdtG/kNNXwcjRXRUCaW8CoGsBSty9VHlmfbDmfhsPRH5nPq7xk3 tt9wbt7QX0ccqFKRoVoz2tuO0/eCafs+RWbcWzYlVcFSyehAtFsjjjmdr7Sju3/X2GxO IA6j+yUKeN5MO9Fura2quWyCNfYi+3xLw+F7zijzNWKTKqvtlYPaYR+3BkRH+13Mi3XD WWyzrjKCBCRWqKHdcg9kp0g0gmM0/n1MAmsxCM+KFzCDwlT9OCXAxfTyj9T2JZlczv43 Yw== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2120.oracle.com with ESMTP id 2uus4kg3n1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Sep 2019 14:36:45 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x86EXR7d079676; Fri, 6 Sep 2019 14:34:44 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserp3020.oracle.com with ESMTP id 2uud7pk6h6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Sep 2019 14:34:44 +0000 Received: from abhmp0009.oracle.com (abhmp0009.oracle.com [141.146.116.15]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x86EWwLi026054; Fri, 6 Sep 2019 14:32:58 GMT Received: from neelam.us.oracle.com (/10.152.128.16) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 06 Sep 2019 07:32:58 -0700 From: Alex Kogan To: linux@armlinux.org.uk, peterz@infradead.org, mingo@redhat.com, will.deacon@arm.com, arnd@arndb.de, longman@redhat.com, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, tglx@linutronix.de, bp@alien8.de, hpa@zytor.com, x86@kernel.org, guohanjun@huawei.com, jglauber@marvell.com Subject: [PATCH v4 5/5] locking/qspinlock: Introduce the shuffle reduction optimization into CNA Date: Fri, 6 Sep 2019 10:25:41 -0400 Message-Id: <20190906142541.34061-6-alex.kogan@oracle.com> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20190906142541.34061-1-alex.kogan@oracle.com> References: <20190906142541.34061-1-alex.kogan@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9372 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1906280000 definitions=main-1909060153 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9372 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1906280000 definitions=main-1909060154 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190906_073714_646597_8866F843 X-CRM114-Status: GOOD ( 16.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alex.kogan@oracle.com, dave.dice@oracle.com, rahul.x.yadav@oracle.com, steven.sistare@oracle.com, daniel.m.jordan@oracle.com Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org This optimization reduces the probability threads will be shuffled between the main and secondary queues when the secondary queue is empty. It is helpful when the lock is only lightly contended. Signed-off-by: Alex Kogan Reviewed-by: Steve Sistare --- kernel/locking/qspinlock_cna.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/kernel/locking/qspinlock_cna.h b/kernel/locking/qspinlock_cna.h index e86182e6163b..1c3a8905b2ca 100644 --- a/kernel/locking/qspinlock_cna.h +++ b/kernel/locking/qspinlock_cna.h @@ -64,6 +64,15 @@ static DEFINE_PER_CPU(u32, seed); #define INTRA_NODE_HANDOFF_PROB_ARG (16) /* + * Controls the probability for enabling the scan of the main queue when + * the secondary queue is empty. The chosen value reduces the amount of + * unnecessary shuffling of threads between the two waiting queues when + * the contention is low, while responding fast enough and enabling + * the shuffling when the contention is high. + */ +#define SHUFFLE_REDUCTION_PROB_ARG (7) + +/* * Return false with probability 1 / 2^@num_bits. * Intuitively, the larger @num_bits the less likely false is to be returned. * @num_bits must be a number between 0 and 31. @@ -230,6 +239,16 @@ static inline void cna_pass_lock(struct mcs_spinlock *node, u32 val = 1; /* + * Limit thread shuffling when the secondary queue is empty. + * This copes with the overhead the shuffling creates when the + * lock is only lightly contended, and threads do not stay + * in the secondary queue long enough to reap the benefit of moving + * them there. + */ + if (node->locked <= 1 && probably(SHUFFLE_REDUCTION_PROB_ARG)) + goto pass_lock; + + /* * Try to find a successor running on the same NUMA node * as the current lock holder. For long-term fairness, * search for such a thread with high probability rather than always. @@ -252,5 +271,6 @@ static inline void cna_pass_lock(struct mcs_spinlock *node, ((struct cna_node *)next_holder)->tail->mcs.next = next; } +pass_lock: arch_mcs_pass_lock(&next_holder->locked, val); } -- 2.11.0 (Apple Git-81) _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel