From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752784AbcELXU5 (ORCPT ); Thu, 12 May 2016 19:20:57 -0400 Received: from mail-by2on0123.outbound.protection.outlook.com ([207.46.100.123]:23883 "EHLO na01-by2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751484AbcELXUz (ORCPT ); Thu, 12 May 2016 19:20:55 -0400 Authentication-Results: hpe.com; dkim=none (message not signed) header.d=none;hpe.com; dmarc=none action=none header.from=hpe.com; Message-ID: <57350FBF.9060807@hpe.com> Date: Thu, 12 May 2016 19:20:31 -0400 From: Waiman Long User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20130109 Thunderbird/10.0.12 MIME-Version: 1.0 To: Waiman Long CC: Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , , , Jiang Liu , Borislav Petkov , Andy Lutomirski , Scott J Norton , Douglas Hatch , Randy Wright Subject: Re: [PATCH v4] x86/hpet: Reduce HPET counter read contention References: <1460486768-34024-1-git-send-email-Waiman.Long@hpe.com> In-Reply-To: <1460486768-34024-1-git-send-email-Waiman.Long@hpe.com> Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [72.71.243.221] X-ClientProxiedBy: BY1PR14CA0042.namprd14.prod.outlook.com (10.161.91.52) To TU4PR84MB0318.NAMPRD84.PROD.OUTLOOK.COM (10.162.186.28) X-MS-Office365-Filtering-Correlation-Id: c545df8a-c630-421d-60a6-08d37abc0f85 X-Microsoft-Exchange-Diagnostics: 1;TU4PR84MB0318;2:tWXcqPN0rT6ocB2FdR4qTMnbigSjebPqUHpToFPv54K5lXv/gnxMtdJR7hS2I8v1sBrxC4npcyhWPc4DtRnQR52lZ6maEiFMXqojYPVE86+m1rGmeJOKISV2sCQWtUhb4I4ssGEtxcK6uPAi6pi/SnFjCnrWwPmBXNKaSxsG1H/Si2vdh20cutirtnLif8as;3:3zNZBrxpbM9Y5O+7mZUx8zEwje2XnRk9rStnF4hW+PYQDABjmgveCYuubRHmUV/2zwFDsrfQUroB94FMvmHyX3Yfw8qzqUy96pMeCUdhGjTupUPrBdlhfFkIfRnRMI9I;25:BVv3ehQS+UwLkp8ZMscW4IE7bdbj72f03gsJ6UMxHT+umKLLi8qk/+NqQ5saCG7a+MRcd4g/FyFex6SVM9u6KAb5ozgZYL/7w/aITTZ2OaAAdzNF30wAfgYAKHMaXmlSyqwatFvIHzBJhsCw66YBCF3uLrh8nqsKPIQNXp9RO21XhrCUlJXo9DQhuWUN1KysklAZoFlDJ1I22ttatn7oWfg1qXOfmt8LvFGVM35sCMr2CyP9Z7T3yBJjmKb/T9g8O6LW/5IeNjH+kWagX23mkMQ2XS0PJ7xwzdOg95miuaLrQ1Me/kSyRBR2zRUrcHmjD3LVVTAOHXYV4aXtQ0rkJlpuiSViAdht+GUZkXvti98gUZSwYZAbLgREibStXQPWoIWvKGYGUQQ6m+Johj+MZ1POacXX0h/PaJ2ZmI/c8VE= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:TU4PR84MB0318; X-LD-Processed: 105b2061-b669-4b31-92ac-24d304d195dc,ExtAddr X-Microsoft-Exchange-Diagnostics: 1;TU4PR84MB0318;20:/uR51042WaZtgpdu/zIz+9i71ueQSqokv+aLQJn25Kn3JfTfYwnM8IO3fT6OjQbyrQzsO2TjUp1zUhWpRewqmhqFg0NTJUbpfWAPULdHepjv+1eF7LITEJWeIz9vcer2JjHtFpo3hi3CbzZHWdVOJDMoaqXpdgvEBXWR48iqbRqVxvMFPmXu2x6Y7Cog74eYCaMhwgUO3PrALj7ndY7BXA5cM2xtgQp20m3FScbW/kypxK6wNJkLVMpeq5VRfLgfLI7NJjd11jTh8wu9OuPInRMa3MzVrkpCXbvxWij91eVmuaqL3rlKMaSItjGwpImLo7gV1bi99gt01OK+oQVD0g==;4:kPudYDEnRRnXlgRoUJPtAnwb4zw4qZE0h3LAZI7WoFsyPjlzr7Wc7C5ZTmWJWp2NvU7BcjuVVEy1n6KxTJ9bUT2Oh/UKCpYVQwdZAt+U76pUnhQx5ijbvtUk9EKDA0P9t9ruXOWH/9wGoQXH8DKFFF5sfYShBXNghWXVG46B4MQzDPoNfbaOPkSIKmIJp2raR+HfPt/KgcDGcOlvnQgSupcS+CYOTq1rjt4ciPbeTSLy2i0yaEHIjWfnehIKArdL35dDo9zX2pMzgICNJ+v4ypEd87slRi/pyWldutS9apyfXl1U5C3Eh8FBQvYeqOluiGq/PW1Z0WuT94YLF8kK12bxov1H705M5aT71NY4aGDgAGOvZozvIC31FI1UfAOc X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001);SRVR:TU4PR84MB0318;BCL:0;PCL:0;RULEID:;SRVR:TU4PR84MB0318; X-Forefront-PRVS: 0940A19703 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(979002)(6049001)(6009001)(377454003)(24454002)(87266999)(36756003)(76176999)(65816999)(83506001)(54356999)(50986999)(3846002)(5008740100001)(6116002)(50466002)(99136001)(230700001)(86362001)(8676002)(64126003)(33656002)(5004730100002)(586003)(77096005)(2950100001)(23756003)(42186005)(6200100001)(117156001)(4001350100001)(59896002)(92566002)(4326007)(189998001)(110136002)(80316001)(19580395003)(81166006)(65956001)(66066001)(47776003)(65806001)(19580405001)(2906002)(969003)(989001)(999001)(1009001)(1019001);DIR:OUT;SFP:1102;SCL:1;SRVR:TU4PR84MB0318;H:[192.168.142.129];FPR:;SPF:None;MLV:ovrnspm;PTR:InfoNoRecords;LANG:en; X-Microsoft-Exchange-Diagnostics: =?iso-8859-1?Q?1;TU4PR84MB0318;23:GRvCCutWMeQ5vaffYAL3nWy8IGpWqoP4JmxLF4X?= =?iso-8859-1?Q?148MUW7WAGEOugSev2is5BsbXHhfwz9XFUIHSWxexQVZLVzKtN9XN6V3jN?= =?iso-8859-1?Q?gPaRu5b9COHBj6XTBgq1RBGdsvsansCupWbJy11mcWaZHf9y2a+IQ0mo1X?= =?iso-8859-1?Q?P13kmPE+rLlbRoY0xmf/6ITVkCPwWxosLIhXSicHiLlxJCKybRPziXN0xP?= =?iso-8859-1?Q?blznn9NVU3KlKOfj8m3qB40R6RjEP1RpeHp/AF/eakB/dvX1JY7iDEx+xJ?= =?iso-8859-1?Q?mF/W0aE8QRXwSar1Ih8aES2xB+ppU5yPnVcs5CR3YSoNOPi12jgTAusZDA?= =?iso-8859-1?Q?u9RI53fRFLbjrBm4fT4E6qaY8buUlTsMedfDu9sHB4LTHismMebmoFITt4?= =?iso-8859-1?Q?Cdk0o/s2K4GLGba/4Jp8dv1c8rro5Sn+PDeGb/hhq1DqPv3YBqaz5q0XyV?= =?iso-8859-1?Q?Yh6CADPdk6dX6t3uSTYaTxYUmjOjjS/vkb+KYxCcDQitQWxCzsQrnDMnjX?= =?iso-8859-1?Q?tgkJGJrcANyQtxeZvIgJGEw7c7cBl73QuWtWS3g8qtHhsuq53ooEXKVeMc?= =?iso-8859-1?Q?1CNN4gtbAqhXVnCFtonWLCPkcMHsKAt0GyAbOUmzQWPY73huxFxmS+B0Ia?= =?iso-8859-1?Q?Oro+7TjteY3K09NRflaaWNGyDiWkZ5RddK4o/3dJ4cCgkHZN2UVOJ9wZB0?= =?iso-8859-1?Q?eYSPpF8197Ph25Ansxow33jsjwZQ1dpJBmMNx2hymzVXDYy7vL7ooN7zRS?= =?iso-8859-1?Q?0wuMAlwih6flDTkDK2QS498ZlzctHI9b1J0t57UCTCSwtkLl8TJ7yrArti?= =?iso-8859-1?Q?x7mXugfzdKaNT+Ky1mmVBcWbAsWSYcHLBGYQ0/pKOduJynkxhP5eHpIRFF?= =?iso-8859-1?Q?5e8UtD8K1UbYPFqRJkWp0jLcDrXW8lr9yFRxeW6HgwLLp9gAJiiz/voq4q?= =?iso-8859-1?Q?mCUj0BxX+lIFvWQZ4LRmtzdfYcr2yc2uZyX+oOujJd6RgXlY8eK7TYPkOS?= =?iso-8859-1?Q?GplXmRnuHrpVkFCRW/tZCsPsOYF0oq5e6VL3DdyE/zImfKjrd7tVoyRsqb?= =?iso-8859-1?Q?470Ykq8drV9gyBBxW7Y/VIcAjTol+0BtFYcb/bSadgyiVKE4PDvGukXVUg?= =?iso-8859-1?Q?1ffPrNhx/74SlVt7kJjn5lweYjcPqHkEPQ/a8M8eu/EsTv+OMO+R2GlYhM?= =?iso-8859-1?Q?JsRGx/+hH1i18ReXljZ5Bo9cwDeOiULloUKSps/k1V86feIJeDdDZcfS+9?= =?iso-8859-1?Q?vLVC8AbstdSEnXU6q2LjW5zvyhGViOsgQsifikIEErn2jQA3a0Q+enUjvQ?= =?iso-8859-1?Q?w3h2GW7w9sFQV9jDv/Cqn2I?= X-Microsoft-Exchange-Diagnostics: 1;TU4PR84MB0318;5:c0FIz490Rvqgfv70PP2fNKmxYYSknGB8bxXz8om77z34DyrZ1VVITDndVIxVG2rb1/Nm2Lk8YiF79MABKI4a1cW882ph97jnPqxWQibWHRB3Wn5bCFouVXvrIKEXFj4B7wOw9tIdSBYLlbcLMtnEcg==;24:vhWOafPeheUtv2oVi/a5WMsyepjR3/njkbgmGyvQkkZu/B/R8utw4wqWEOg1rkbQIZz+PuTofN+gIrQCeINC34IGOTQU4AWOgmxtjaOPIHk=;7:MAAwl9GZ49fjdy7UceFK91YFHt0qsPSlFANspLXFbqT8pS9N7+Zr7bpSBRLbJc5ZHz62UZ5Riox0MVwjLnqkKwECIwHd87niVd15PQ6ThXD5xxzmY2vHGPDqH/4JTi0rAKsxPIwW3IqT5BhEwys3enVUH/b4C+c/5CtPQ7lFj4VrOkToxutrrLBYp3oNGsYY SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: hpe.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 May 2016 23:20:47.4240 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: TU4PR84MB0318 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 04/12/2016 02:46 PM, Waiman Long wrote: > On a large system with many CPUs, using HPET as the clock source can > have a significant impact on the overall system performance because > of the following reasons: > 1) There is a single HPET counter shared by all the CPUs. > 2) HPET counter reading is a very slow operation. > > Using HPET as the default clock source may happen when, for example, > the TSC clock calibration exceeds the allowable tolerance. Something > the performance slowdown can be so severe that the system may crash > because of a NMI watchdog soft lockup, for example. > > This patch attempts to reduce HPET read contention by using the fact > that if more than one CPUs are trying to access HPET at the same time, > it will be more efficient if one CPU in the group reads the HPET > counter and shares it with the rest of the group instead of each > group member reads the HPET counter individually. > > This is done by using a combination word with a sequence number and > a bit lock. The CPU that gets the bit lock will be responsible for > reading the HPET counter and update the sequence number. The others > will monitor the change in sequence number and grab the HPET counter > accordingly. This change is enabled on SMP configuration. > > On a 4-socket Haswell-EX box with 72 cores (HT off), running the > AIM7 compute workload (1500 users) on a 4.6-rc1 kernel (HZ=1000) > with and without the patch has the following performance numbers > (with HPET or TSC as clock source): > > TSC = 646515 jobs/min > HPET w/o patch = 566708 jobs/min > HPET with patch = 638791 jobs/min > > The perf profile showed a reduction of the %CPU time consumed by > read_hpet from 4.99% without patch to 1.41% with patch. > > On a 16-socket IvyBridge-EX system with 240 cores (HT on), on the > other hand, the performance numbers of the same benchmark were: > > TSC = 3145329 jobs/min > HPET w/o patch = 1108537 jobs/min > HPET with patch = 3019934 jobs/min > > The corresponding perf profile showed a drop of CPU consumption of > the read_hpet function from more than 34% to just 2.96%. > > Signed-off-by: Waiman Long > --- > v3->v4: > - Move hpet_save inside the CONFIG_SMP block to fix a compilation > warning in non-SMP build. > > v2->v3: > - Make the hpet optimization the default for SMP configuration. So > no documentation change is needed. > - Remove threshold checking code as it should not be necessary and > can be potentially unsafe. > > v1->v2: > - Reduce the CPU threshold to 32. > - Add a kernel parameter to explicitly enable or disable hpet > optimization. > - Change hpet_save.hpet type to u32 to make sure that read& write > is atomic on i386. > > arch/x86/kernel/hpet.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++++ > 1 files changed, 84 insertions(+), 0 deletions(-) > > I haven't received any feedback on this patch since mid-April. I would like to know if the current patch is good enough or some additional changes are still needed to make it merge-able upstream. Thanks, Longman