From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935542AbcIFPdb (ORCPT ); Tue, 6 Sep 2016 11:33:31 -0400 Received: from mx1.redhat.com ([209.132.183.28]:39238 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932180AbcIFPd3 (ORCPT ); Tue, 6 Sep 2016 11:33:29 -0400 Message-ID: <57CEE1C7.40505@redhat.com> Date: Tue, 06 Sep 2016 11:33:27 -0400 From: Prarit Bhargava User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Waiman Long , Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" CC: linux-kernel@vger.kernel.org, x86@kernel.org, Borislav Petkov , Andy Lutomirski , Dave Hansen , Scott J Norton , Douglas Hatch , Randy Wright Subject: Re: [RESEND PATCH v6] x86/hpet: Reduce HPET counter read contention References: <1473175676-27713-1-git-send-email-Waiman.Long@hpe.com> In-Reply-To: <1473175676-27713-1-git-send-email-Waiman.Long@hpe.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Tue, 06 Sep 2016 15:33:29 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 09/06/2016 11:27 AM, Waiman Long wrote: > On a large system with many CPUs, using HPET as the clock source can > have a significant impact on the overall system performance because > of the following reasons: > 1) There is a single HPET counter shared by all the CPUs. > 2) HPET counter reading is a very slow operation. > > Using HPET as the default clock source may happen when, for example, > the TSC clock calibration exceeds the allowable tolerance. Something > the performance slowdown can be so severe that the system may crash > because of a NMI watchdog soft lockup, for example. > > During the TSC clock calibration process, the default clock source > will be set temporarily to HPET. For systems with many CPUs, it is > possible that NMI watchdog soft lockup may occur occasionally during > that short time period where HPET clocking is active as is shown in > the kernel log below: > > [ 71.618132] NetLabel: Initializing > [ 71.621967] NetLabel: domain hash size = 128 > [ 71.626848] NetLabel: protocols = UNLABELED CIPSOv4 > [ 71.632418] NetLabel: unlabeled traffic allowed by default > [ 71.638679] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 > [ 71.646504] hpet0: 8 comparators, 64-bit 14.318180 MHz counter > [ 71.655313] Switching to clocksource hpet > [ 95.679135] BUG: soft lockup - CPU#144 stuck for 23s! [swapper/144:0] > [ 95.693363] BUG: soft lockup - CPU#145 stuck for 23s! [swapper/145:0] > [ 95.694203] Modules linked in: > [ 95.694697] CPU: 145 PID: 0 Comm: swapper/145 Not tainted 3.10.0-327.el7.x86_64 #1 > [ 95.695580] BUG: soft lockup - CPU#582 stuck for 23s! [swapper/582:0] > [ 95.696145] Hardware name: HP Superdome2 16s x86, BIOS Bundle: 008.001.006 SFW: 041.063.152 01/16/2016 > [ 95.698128] BUG: soft lockup - CPU#357 stuck for 23s! [swapper/357:0] > > This patch attempts to address the above issues by reducing HPET read > contention using the fact that if more than one CPUs are trying to > access HPET at the same time, it will be more efficient when only > one CPU in the group reads the HPET counter and shares it with the > rest of the group instead of each group member trying to read the > HPET counter individually. > > This is done by using a combination word with a sequence number and > a bit lock. The CPU that gets the bit lock will be responsible for > reading the HPET counter and update the sequence number. The others > will monitor the change in sequence number and grab the HPET counter > value accordingly. This change is only enabled on SMP configuration. > > On a 4-socket Haswell-EX box with 144 threads (HT on), running the > AIM7 compute workload (1500 users) on a 4.8-rc1 kernel (HZ=1000) > with and without the patch has the following performance numbers > (with HPET or TSC as clock source): > > TSC = 1042431 jobs/min > HPET w/o patch = 798068 jobs/min > HPET with patch = 1029445 jobs/min > > The perf profile showed a reduction of the %CPU time consumed by > read_hpet from 11.19% without patch to 1.24% with patch. > > Signed-off-by: Waiman Long This resolves the boot-time problems on my systems. I've also seen a performance increase of about 5% with this patch when using the HPET. Tested-by: Prarit Bhargava P.