From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753706AbcEJC0I (ORCPT ); Mon, 9 May 2016 22:26:08 -0400 Received: from mail-bn1bon0134.outbound.protection.outlook.com ([157.56.111.134]:22848 "EHLO na01-bn1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750982AbcEJC0G (ORCPT ); Mon, 9 May 2016 22:26:06 -0400 Authentication-Results: hp.com; dkim=none (message not signed) header.d=none;hp.com; dmarc=none action=none header.from=hpe.com; Message-ID: <573146B2.2010807@hpe.com> Date: Mon, 9 May 2016 22:25:54 -0400 From: Waiman Long User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20130109 Thunderbird/10.0.12 MIME-Version: 1.0 To: Jason Low CC: Peter Zijlstra , Ingo Molnar , , Davidlohr Bueso , Scott J Norton , , Jason Low Subject: Re: [PATCH] locking/rwsem: Optimize write lock slowpath References: <1462821397.2701.16.camel@j-VirtualBox> In-Reply-To: <1462821397.2701.16.camel@j-VirtualBox> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [63.156.242.50] X-ClientProxiedBy: BY1PR15CA0004.namprd15.prod.outlook.com (10.162.17.142) To DF4PR84MB0316.NAMPRD84.PROD.OUTLOOK.COM (10.162.193.30) X-MS-Office365-Filtering-Correlation-Id: d5d876d6-1f90-4701-f037-08d3787a6ed5 X-Microsoft-Exchange-Diagnostics: 1;DF4PR84MB0316;2:/LNoWhCP/fp0nVN35Xf9YEN08WeC9CbDLN4Q3nEq0E01xPs+L3Ps0hOs+jngXlNuXUIjx17l5SMwpZCkMYEU177xtNZ6C2MjZlNq8ZmNxK3MIkTF6lCMSJeK3AYZVYRXkjE3nXs6Dto7blF2ohsVnHwsnhXAJSNO2UMe3gdKGeSER0O4ed0NhbDoL8kTy7Mo;3:/OSl4F/J1N/t3pu7iIRUg9lIHVmnxJAFEInAOQlYslP3u1hRv8tQXHXOgtnwYvSemDEDw7Q70JvAVe+KqZ9EHsa42ahkld6Z7TxfuKPtjDoxDZtvFD+ZZv4734yJFkoM X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:DF4PR84MB0316; X-Microsoft-Exchange-Diagnostics: 1;DF4PR84MB0316;25:PJIhsxNZXPljibbZl+quQ/FmK+cXtfqzEee6i6ptKaEAlcxXMaLqNsZf2JHisf07xwFP6fO3sM8q2kgZM9VgD+9QGtP/gUebL6L4PRAxbT2vUWLnHR8Y2Gh4iexd7YQvO/upOTg/3b1CaSnjYoxBzON6DccEomODyk4+cx7YVygt8viKr3GEsogugKCVjvv149QhEBFPdooCo+DeAP6nW+PF/wMHtHhwyz5++g7UXlmWG4bOyvcku0f2Avj8N8MLEtWy03oBhgTJCK3vVGmtyH+RbwwkZfXjx9OxtdzZweyyjHsfD4KwaNOe24eZukD1P8wQ/LbJUKssQWjf9f5VTYdXUOuAqFw3WXmWOMuyY/JVQXbujMFAnXpVAnCek0SB4M0Af/Z2SxKtq6ojHiBE37OWgSorBMU1mH0liku9VYxiLlPyRU+1rELmglssQp28vZmfE3Qs7Hd8vnQCmyQ29ZiAOYKa8qLf+ltgnaxYAKbRN1cRrgKittmysWJ/mzJaywpVC2JPyQGEBm6k4zHdOkiB0jdJoq5WcWPTc9WRjWhYSy5VMniQdRLowdzfW1MeZ96ti5+F3JPOImGVv0nJVrSRIMGjNaDoHB92Q1hw273X+aTzPJkOzyAh76FS9dT2Lj+gJCSpmzKXwXD2hBPSpxb0GddFVY02SIvyTSUZtTzHuUThGlRHoVZqqaDGWNzStt0w9mb1qDNgdVtBjizzGwsIP2oSa1f08NcT+hRgE7cxcStNz1gYHz6QUrwY9o/nLTK+SuV4DjYU4OOnrpMIwA== X-LD-Processed: 105b2061-b669-4b31-92ac-24d304d195dc,ExtAddr X-Microsoft-Exchange-Diagnostics: 1;DF4PR84MB0316;20:zSnbeLgmHNAO89XhoRHwn5Db9qzaJamwH7IRhkGIg34Lo/IB3BkPb72qqRGxJMogFYPzH2p/vSvu1rdZqY8ZEQ/mS15ioLWatxMwTjB+b21UdN+jYebf+CrBNk5lET4LE7pTuilanO7LbA/uzfzZ6dBh3qb00gBF8lgRPS1xkF4FEFYXvW8+TFaovyImap87ed6FcTMFrzpyuA6y8EuiYMi893FbxdsxsgEckHWUk6urCRFJIhuyJPunfUH8BSjty+C1mfkd1psqL3PbAjt3tm9pKo8ndpvlwSVh5NwiY+Wo4dUxj9YY7q9kRS/dfH9fE7pqh2IejnfYNs7YlHMfxQ==;4:KHsg3AXz/sXicKGq3Gsr0NXqQGpgWGMhuF3Ks8JyXgT+YIjVztAJmC+7rwq7miLQHKqRfcy7n1BPXlsl0Fd+yRvZ2oIYhqlPcUvaFAElP1zPOuVkYzK8+h01JicLepBjIMwN1cIdnkPKpPTddhdfhCc6KLstorqLY03F9jDFjSamh/sYBLFFV8CIy2/mVK/2NDdf7PX3X8xYOYs1xrSOCSGKNSK8JHfyRkJm2qupg2+dkEfbeqBwZvk29cdxp3X/5u5Ux+oszNqTzBnbplvXSHDYED986OR9ubuNnxnoHIN8MK/KA2oIIfRvF27537CzMoIxPy28ILdRO3esbzVgLRfuQ7Y5dpSue9N6zhydNoQ8lVZtx88v/S/MndX0H5TG X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001);SRVR:DF4PR84MB0316;BCL:0;PCL:0;RULEID:;SRVR:DF4PR84MB0316; X-Forefront-PRVS: 0938781D02 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(6049001)(6009001)(24454002)(377454003)(87266999)(4326007)(42186005)(54356999)(76176999)(117156001)(5008740100001)(65816999)(50986999)(64126003)(47776003)(230700001)(77096005)(2950100001)(4001350100001)(110136002)(189998001)(50466002)(65806001)(66066001)(33656002)(65956001)(36756003)(5004730100002)(6116002)(3846002)(19580405001)(19580395003)(81166005)(86362001)(99136001)(23676002)(2906002)(80316001)(83506001)(586003)(92566002)(7059030);DIR:OUT;SFP:1102;SCL:1;SRVR:DF4PR84MB0316;H:[192.168.142.128];FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtERjRQUjg0TUIwMzE2OzIzOlRwa1ZzcWx0YlJVL2labGIrTFZ6ZVFWUFlq?= =?utf-8?B?QlQwREV6aFB0Vlo3T2c5R3Z1VE51VkVwSlNjcloyckEvOGFwRklsMmpmdllC?= =?utf-8?B?eGtnaVFvSGYxTGV5bWdxRUp5V05JeHdUU0ZnT1hVYm5GZ1NtWmsxWWlaYlRw?= =?utf-8?B?czZ0c0RuNFdEV09FYjI4bG5pWjhqYTVrN1FtNEFCcko4UGo3dUdHQXJ1MmtW?= =?utf-8?B?dWJsYmpYeDN4QXBHOC9UYlVhNGtvTXVnVis0NUVGRjZjWE01R3JqQlpPYTJW?= =?utf-8?B?YjdWY01qQUdLeW9qR0kxaEJVaDF0ck1oWFFjWkFaeTlQSUZodGx5YVlBRUhR?= =?utf-8?B?ZGhyYXoybWFheE1xQ2poaDBPd2Q0Z0lmeFh1MDI2MFAwSzduNEZLTy9ySXRq?= =?utf-8?B?Y2NjVWxVTk5POUwzbTJBYm95UEhHWHlRY3pTT2RQYTI2WHY4ZnBwOEFLbmVv?= =?utf-8?B?N0V6ZGZYYmZxczBWUlM0OTk1bDdLVUdOaWUxTXdGaWQyWWUvUkJ4UGN1dHMr?= =?utf-8?B?Z1NvSUp3cUlNUVAybHB2bThRLy9BQjJGc2ZNSWVHWjAzK3FiQjFzbUg5Ymov?= =?utf-8?B?cVMrQVBnYmRWZDJXK091QVIrSDJDTDR3UVlJNGxYZkk0R3lSb2ZvVk9zV05s?= =?utf-8?B?YXpLZStyR3pCKzFaVTZZQzQ0QXVTUWVBdXBadVVyM1grbFJsMUpUd1ROMFAz?= =?utf-8?B?UE9scDYvSkJydk5nZ0VhSFl6d3dVKzU1QStHN2Z5NFNzblprK01JMXVIQUVH?= =?utf-8?B?ckVXZlR1eE52RHR1dkNZNWd4aHd2QjRtVGlOdWp6RVh5NWJwaDVFTUhySE1p?= =?utf-8?B?WDBoK2NwUXVacHFEM2hZNnQ1TWhkZHdqVThTUGpTSkMwcnBVdVh3RXF2MVk2?= =?utf-8?B?bnhONVRzS2F1eVJJanZOYWdmRGhoZDJzMHk3UVBEYVU2bHJiK3hWYU9GNGND?= =?utf-8?B?R09zWlRKd1U0clJlT2RnSnVBUEU3djBIWmJJbUxxOUZaM2ErQ01ZV29oL0Rr?= =?utf-8?B?ZnZvbGROeHhHREZLOG1KVjVlK0F1cXVoRXpLeVpadWRSQlNXZWd0YWhsVTda?= =?utf-8?B?aE1rMFFGbmJoaThRMjZ5QnJFQ2x3NDNZUlRDWDFRYkZNdUYrZlA4U0ZpSDZs?= =?utf-8?B?ZlNyRTVnTkliL2h1Q1RWWlBmR3hPT3lrQ095dnNaYXR3eUZLbkUrNHJSaENo?= =?utf-8?B?b3lLYzZHZTRMc3JqVGJCZzJ0UitRVUxlbUF3RDRQeStMVGFRTUtZSzhNRnNz?= =?utf-8?B?U3VIWDdnMWNIemRZTWhYTFFuczVkb050VW1WLy81NFVyU0ozeUx2SHROVTZG?= =?utf-8?B?UFlXbldpbkhxUWw5ZVdzTGlqQ3lFejVQeWdHcjA0c0F3SlpPV0RXSnp0MW9p?= =?utf-8?B?MmcvL1hOemYvWDIra2wydjIyWDNGRWFFMU9vMmIxMm8xT1JPOWJ5TTB3TTV4?= =?utf-8?B?QlgwSXVldzZFTy95TmIxeEJGaW5MWXF2OWpldHJET093aTRzRHRHOCtvaWx5?= =?utf-8?Q?heT1PL6mLtJS78S23cNkdAZXI=3D?= X-Microsoft-Exchange-Diagnostics: 1;DF4PR84MB0316;5:/NCNAsXyci14wzTBR0+LWMZw53Mt27swuLjJrm6UFDhQWitrqBalRKV2VY9ROQagpXjHKOwOyu0WDQ6oPCGGKdJ6tcv+ZR8W//cmm0LOK1uPn0ugg1TnXeEMJNkERqvH9Hv2e0x3u5T44U1+FtTWTA==;24:2PcXgqcZ4djqoCMgt9/j64WeRxynKeZ1Ic292cQVFawXhqz0F4SGOAqHOPbQzjF9vXpmE6MBucfT3WcOoQF9TINoiX3g/XADo9zUwuD+eIY=;7:zK/g0Xn5u+dbrrC2E31vQ22OZdIC4Qiu4GMf38LKkP0HO6de6NTw2oJkuskCIz8zUU1gE8GvtDtgfqEV9uj+dFfPjFQKubwwtW8/781o+Lp1Ms55qPC/haGcDN5VPEMstz+UKfxkeSWkJoDqBiOklkbhwO09pPn/RC/GIF1A6Ig+/bsZLy69Glvf4lrNEHLB SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: hpe.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 May 2016 02:26:01.7729 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: DF4PR84MB0316 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 05/09/2016 03:16 PM, Jason Low wrote: > When acquiring the rwsem write lock in the slowpath, we first try > to set count to RWSEM_WAITING_BIAS. When that is successful, > we then atomically add the RWSEM_WAITING_BIAS in cases where > there are other tasks on the wait list. This causes write lock > operations to often issue multiple atomic operations. > > We can instead make the list_is_singular() check first, and then > set the count accordingly, so that we issue at most 1 atomic > operation when acquiring the write lock and reduce unnecessary > cacheline contention. > > Signed-off-by: Jason Low > --- > kernel/locking/rwsem-xadd.c | 20 +++++++++++++------- > 1 file changed, 13 insertions(+), 7 deletions(-) > > diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c > index df4dcb8..23c33e6 100644 > --- a/kernel/locking/rwsem-xadd.c > +++ b/kernel/locking/rwsem-xadd.c > @@ -258,14 +258,20 @@ EXPORT_SYMBOL(rwsem_down_read_failed); > static inline bool rwsem_try_write_lock(long count, struct rw_semaphore *sem) > { > /* > - * Try acquiring the write lock. Check count first in order > - * to reduce unnecessary expensive cmpxchg() operations. > + * Avoid trying to acquire write lock if count isn't RWSEM_WAITING_BIAS. > */ > - if (count == RWSEM_WAITING_BIAS&& > - cmpxchg_acquire(&sem->count, RWSEM_WAITING_BIAS, > - RWSEM_ACTIVE_WRITE_BIAS) == RWSEM_WAITING_BIAS) { > - if (!list_is_singular(&sem->wait_list)) > - rwsem_atomic_update(RWSEM_WAITING_BIAS, sem); > + if (count != RWSEM_WAITING_BIAS) > + return false; > + > + /* > + * Acquire the lock by trying to set it to ACTIVE_WRITE_BIAS. If there > + * are other tasks on the wait list, we need to add on WAITING_BIAS. > + */ > + count = list_is_singular(&sem->wait_list) ? > + RWSEM_ACTIVE_WRITE_BIAS : > + RWSEM_ACTIVE_WRITE_BIAS + RWSEM_WAITING_BIAS; > + > + if (cmpxchg_acquire(&sem->count, RWSEM_WAITING_BIAS, count) == RWSEM_WAITING_BIAS) { > rwsem_set_owner(sem); > return true; > } Acked-by: Waiman Long