From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20D12C43460 for ; Tue, 27 Apr 2021 01:14:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EA44F6127A for ; Tue, 27 Apr 2021 01:14:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235752AbhD0BOn (ORCPT ); Mon, 26 Apr 2021 21:14:43 -0400 Received: from mga05.intel.com ([192.55.52.43]:37532 "EHLO mga05.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232022AbhD0BOm (ORCPT ); Mon, 26 Apr 2021 21:14:42 -0400 IronPort-SDR: Ku/WFkZcRW6NR1Dj1BY6G0A8dMlh6nY5eR3kUXlJp79c724kyVpvMHVVF0vrijJp5EXIQzWgFH TBRYnMVXYC5w== X-IronPort-AV: E=McAfee;i="6200,9189,9966"; a="281760714" X-IronPort-AV: E=Sophos;i="5.82,252,1613462400"; d="scan'208";a="281760714" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Apr 2021 18:14:00 -0700 IronPort-SDR: 10PoMiAItFcTT27cVjl3secmZUDE4uIq3Ld7KlDidgNMbYlBtY0Kk9CxOVW6Jn0NZULdZGRaZg VHTp4mPBAVrg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.82,252,1613462400"; d="scan'208";a="429606418" Received: from shbuild999.sh.intel.com (HELO localhost) ([10.239.147.94]) by orsmga008.jf.intel.com with ESMTP; 26 Apr 2021 18:13:56 -0700 Date: Tue, 27 Apr 2021 09:13:55 +0800 From: Feng Tang To: "Paul E. McKenney" Cc: tglx@linutronix.de, linux-kernel@vger.kernel.org, john.stultz@linaro.org, sboyd@kernel.org, corbet@lwn.net, Mark.Rutland@arm.com, maz@kernel.org, kernel-team@fb.com, neeraju@codeaurora.org, ak@linux.intel.com, zhengjun.xing@intel.com, Xing Zhengjun Subject: Re: [PATCH v10 clocksource 6/7] clocksource: Forgive tsc_early pre-calibration drift Message-ID: <20210427011355.GC89018@shbuild999.sh.intel.com> References: <20210425224540.GA1312438@paulmck-ThinkPad-P17-Gen-1> <20210425224709.1312655-6-paulmck@kernel.org> <20210426150127.GB23119@shbuild999.sh.intel.com> <20210426152529.GX975577@paulmck-ThinkPad-P17-Gen-1> <20210426153605.GB89018@shbuild999.sh.intel.com> <20210426182652.GE975577@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210426182652.GE975577@paulmck-ThinkPad-P17-Gen-1> User-Agent: Mutt/1.5.24 (2015-08-30) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Apr 26, 2021 at 11:26:52AM -0700, Paul E. McKenney wrote: > On Mon, Apr 26, 2021 at 11:36:05PM +0800, Feng Tang wrote: > > On Mon, Apr 26, 2021 at 08:25:29AM -0700, Paul E. McKenney wrote: > > > On Mon, Apr 26, 2021 at 11:01:27PM +0800, Feng Tang wrote: > > > > Hi Paul, > > > > > > > > On Sun, Apr 25, 2021 at 03:47:07PM -0700, Paul E. McKenney wrote: > > > > > Because the x86 tsc_early clocksource is given a quick and semi-accurate > > > > > calibration (by design!), it might have drift rates well in excess of > > > > > the 0.1% limit that is in the process of being adopted. > > > > > > > > > > Therefore, add a max_drift field to the clocksource structure that, when > > > > > non-zero, specifies the maximum allowable drift rate in nanoseconds over > > > > > a half-second period. The tsc_early clocksource initializes this to five > > > > > miliseconds, which corresponds to the 1% drift rate limit suggested by > > > > > Xing Zhengjun. This max_drift field is intended only for early boot, > > > > > so clocksource_watchdog() splats if it encounters a non-zero value in > > > > > this field more than 60 seconds after boot, inspired by a suggestion by > > > > > Thomas Gleixner. > > > > > > > > > > This was tested by setting the clocksource_tsc ->max_drift field to 1, > > > > > which, as expected, resulted in a clock-skew event. > > > > > > > > We've run the same last for this v10, and those 'unstable' thing [1] can > > > > not be reproduced! > > > > > > Good to hear! ;-) > > > > > > > We've reported one case that tsc can be wrongly judged as 'unstable' > > > > by 'refined-jiffies' watchdog [1], while reducing the threshold could > > > > make it easier to be triggered. > > > > > > > > It could be reproduced on the a plaform with a 115200 serial console, > > > > and hpet been disabled (several x86 platforms has this), add > > > > 'initcall_debug' cmdline parameter to get more debug message, we can > > > > see: > > > > > > > > [ 1.134197] clocksource: timekeeping watchdog on CPU1: Marking clocksource 'tsc-early' as unstable because the skew is too large: > > > > [ 1.134214] clocksource: 'refined-jiffies' wd_nesc: 500000000 wd_now: ffff8b35 wd_last: ffff8b03 mask: ffffffff > > > > [ 1.134217] clocksource: 'tsc-early' cs_nsec: 507537855 cs_now: 4e63c9d09 cs_last: 4bebd81f5 mask: ffffffffffffffff > > > > [ 1.134220] clocksource: No current clocksource. > > > > [ 1.134222] tsc: Marking TSC unstable due to clocksource watchdog > > > > > > Just to make sure I understand: "could be reproduced" as in this is the > > > result from v9, and v10 avoids this, correct? > > > > Sorry I didn't make it clear. This is a rarely happened case, and can > > be reproduced with upstream kerenl, which has 62.5 ms threshold. 6/7 & > > 7/7 patch of reducing the threshold can make it easier to be triggered. > > Ah, OK, so this could be considered to be a benefit of this series, then. > > Does this happen only for tsc-early, or for tsc as well? > > Has it already been triggered on v10 of this series? (I understand that > it certainly should be easier to trigger, just curious whether this has > already happened.) Yes, it has. The upper log is from v10 (actually it's the 'dev' branch of your linux-rcu git, which I didn't find obvious difference) on a client platform [ 1.134214] clocksource: 'refined-jiffies' wd_nesc: 500000000 wd_now: ffff8b35 wd_last: ffff8b03 mask: ffffffff [ 1.134217] clocksource: 'tsc-early' cs_nsec: 507537855 cs_now: 4e63c9d09 cs_last: 4bebd81f5 mask: ffffffffffffffff The deviation is 7537855 ns (7.5 ms). And as said before, it needs many pre-conditions to be triggered. Also I found the debug patch is useful, which prints out the direct nanoseconds info when 'unstable' is detected. kernel/time/clocksource.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c index a374cf7b6336..5370f0c84981 100644 --- a/kernel/time/clocksource.c +++ b/kernel/time/clocksource.c @@ -443,10 +443,10 @@ static void clocksource_watchdog(struct timer_list *unused) if (abs(cs_nsec - wd_nsec) > md) { pr_warn("timekeeping watchdog on CPU%d: Marking clocksource '%s' as unstable because the skew is too large:\n", smp_processor_id(), cs->name); - pr_warn(" '%s' wd_now: %llx wd_last: %llx mask: %llx\n", - watchdog->name, wdnow, wdlast, watchdog->mask); - pr_warn(" '%s' cs_now: %llx cs_last: %llx mask: %llx\n", - cs->name, csnow, cslast, cs->mask); + pr_warn(" '%s' wd_nesc: %lld wd_now: %llx wd_last: %llx mask: %llx\n", + watchdog->name, wd_nsec, wdnow, wdlast, watchdog->mask); + pr_warn(" '%s' cs_nsec: %lld cs_now: %llx cs_last: %llx mask: %llx\n", + cs->name, cs_nsec, csnow, cslast, cs->mask); if (curr_clocksource == cs) pr_warn(" '%s' is current clocksource.\n", cs->name); else if (curr_clocksource) -- 2.27.0 Thanks, Feng > > Thanx, Paul