From: John Stultz <john.stultz@linaro.org>
To: lkml <linux-kernel@vger.kernel.org>
Cc: Shaohua Li <shli@fb.com>, Prarit Bhargava <prarit@redhat.com>,
Richard Cochran <richardcochran@gmail.com>,
Daniel Lezcano <daniel.lezcano@linaro.org>,
Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@kernel.org>,
John Stultz <john.stultz@linaro.org>
Subject: [PATCH 8/9] clocksource: Improve unstable clocksource detection
Date: Mon, 17 Aug 2015 13:41:02 -0700 [thread overview]
Message-ID: <1439844063-7957-9-git-send-email-john.stultz@linaro.org> (raw)
In-Reply-To: <1439844063-7957-1-git-send-email-john.stultz@linaro.org>
From: Shaohua Li <shli@fb.com>
>From time to time we saw TSC is marked as unstable in our systems, while
the CPUs declare to have stable TSC. Looking at the clocksource unstable
detection, there are two problems:
- watchdog clock source wrap. HPET is the most common watchdog clock
source. It's 32-bit and runs in 14.3Mhz. That means the hpet counter
can wrap in about 5 minutes.
- threshold isn't scaled against interval. The threshold is 0.0625s in
0.5s interval. What if the actual interval is bigger than 0.5s?
The watchdog runs in a timer bh, so hard/soft irq can defer its running.
Heavy network stack softirq can hog a cpu. IPMI driver can disable
interrupt for a very long time. The first problem is mostly we are
suffering I think.
Here is a simple patch to fix the issues. If the waterdog doesn't run
for a long time, we ignore the detection. This should work for the two
problems. For the second one, we probably doen't need to scale if the
interval isn't very long.
Cc: Prarit Bhargava <prarit@redhat.com>
Cc: Richard Cochran <richardcochran@gmail.com>
Cc: Daniel Lezcano <daniel.lezcano@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Shaohua Li <shli@fb.com>
Signed-off-by: John Stultz <john.stultz@linaro.org>
---
kernel/time/clocksource.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c
index 841b72f..8417c83 100644
--- a/kernel/time/clocksource.c
+++ b/kernel/time/clocksource.c
@@ -122,9 +122,10 @@ static int clocksource_watchdog_kthread(void *data);
static void __clocksource_change_rating(struct clocksource *cs, int rating);
/*
- * Interval: 0.5sec Threshold: 0.0625s
+ * Interval: 0.5sec MaxInterval: 1s Threshold: 0.0625s
*/
#define WATCHDOG_INTERVAL (HZ >> 1)
+#define WATCHDOG_MAX_INTERVAL_NS (NSEC_PER_SEC)
#define WATCHDOG_THRESHOLD (NSEC_PER_SEC >> 4)
static void clocksource_watchdog_work(struct work_struct *work)
@@ -217,7 +218,9 @@ static void clocksource_watchdog(unsigned long data)
continue;
/* Check the deviation from the watchdog clocksource. */
- if ((abs(cs_nsec - wd_nsec) > WATCHDOG_THRESHOLD)) {
+ if ((abs(cs_nsec - wd_nsec) > WATCHDOG_THRESHOLD) &&
+ cs_nsec < WATCHDOG_MAX_INTERVAL_NS &&
+ wd_nsec < WATCHDOG_MAX_INTERVAL_NS) {
pr_warn("timekeeping watchdog: Marking clocksource '%s' as unstable because the skew is too large:\n",
cs->name);
pr_warn(" '%s' wd_now: %llx wd_last: %llx mask: %llx\n",
--
1.9.1
next prev parent reply other threads:[~2015-08-17 20:41 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-08-17 20:40 [PATCH 0/9] Time items for 4.3 John Stultz
2015-08-17 20:40 ` [PATCH 1/9] timer_list: Add the base offset so remaining nsecs are accurate for non monotonic timers John Stultz
2015-08-17 21:01 ` Shuah Khan
2015-08-17 21:04 ` Shuah Khan
2015-08-17 21:05 ` John Stultz
2015-08-17 20:40 ` [PATCH 2/9] time: Fix nanosecond file time rounding in timespec_trunc() John Stultz
2015-08-17 22:14 ` Thomas Gleixner
2015-08-17 20:40 ` [PATCH 3/9] time: Always make sure wall_to_monotonic isn't positive John Stultz
2015-08-17 20:40 ` [PATCH 4/9] time: Add the common weak version of update_persistent_clock() John Stultz
2015-08-17 20:40 ` [PATCH 5/9] time: Introduce struct itimerspec64 John Stultz
2015-08-17 20:41 ` [PATCH 6/9] time: Introduce current_kernel_time64() John Stultz
2015-08-17 20:41 ` [PATCH 7/9] time: Introduce timespec64_to_jiffies()/jiffies_to_timespec64() John Stultz
2015-08-17 20:41 ` John Stultz [this message]
2015-08-17 22:04 ` [PATCH 8/9] clocksource: Improve unstable clocksource detection Thomas Gleixner
2015-08-17 22:17 ` John Stultz
2015-08-18 2:57 ` Shaohua Li
2015-08-18 3:39 ` John Stultz
2015-08-18 8:57 ` Thomas Gleixner
2015-08-18 8:38 ` Thomas Gleixner
2015-08-18 17:49 ` John Stultz
2015-08-18 19:28 ` Thomas Gleixner
2015-08-18 20:11 ` John Stultz
2015-08-18 20:18 ` Thomas Gleixner
2015-08-26 17:15 ` Shaohua Li
2015-08-31 21:12 ` Shaohua Li
2015-08-31 21:47 ` Thomas Gleixner
2015-08-31 22:39 ` Shaohua Li
2015-09-01 17:13 ` Thomas Gleixner
2015-09-01 18:14 ` Shaohua Li
2015-09-01 18:55 ` Thomas Gleixner
2015-09-01 19:35 ` Steven Rostedt
2015-09-02 6:50 ` Peter Zijlstra
2015-08-17 20:41 ` [PATCH 9/9] clocksource: Sanity check watchdog clocksource John Stultz
2015-08-17 21:24 ` Thomas Gleixner
2015-08-17 22:03 ` John Stultz
2015-08-17 22:08 ` Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1439844063-7957-9-git-send-email-john.stultz@linaro.org \
--to=john.stultz@linaro.org \
--cc=daniel.lezcano@linaro.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=prarit@redhat.com \
--cc=richardcochran@gmail.com \
--cc=shli@fb.com \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox