From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755645AbcDGJtR (ORCPT ); Thu, 7 Apr 2016 05:49:17 -0400 Received: from mail-bn1bon0115.outbound.protection.outlook.com ([157.56.111.115]:57075 "EHLO na01-bn1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751346AbcDGJtO (ORCPT ); Thu, 7 Apr 2016 05:49:14 -0400 X-Greylist: delayed 5671 seconds by postgrey-1.27 at vger.kernel.org; Thu, 07 Apr 2016 05:49:14 EDT Authentication-Results: gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=ni.com; References: <1458067825-11880-1-git-send-email-gratian.crisan@ni.com> User-agent: mu4e 0.9.13; emacs 24.5.50.1 From: Gratian Crisan To: John Stultz CC: Gratian Crisan , Thomas Gleixner , lkml , Gratian Crisan Subject: Re: [PATCH RFC] clocksource: Detect a watchdog overflow In-Reply-To: Date: Thu, 7 Apr 2016 01:14:31 -0700 Message-ID: <877fg9n8d4.fsf@ni.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [209.65.105.133] X-ClientProxiedBy: DM2PR10CA0007.namprd10.prod.outlook.com (25.160.213.17) To BN1PR04MB857.namprd04.prod.outlook.com (10.255.204.152) X-MS-Office365-Filtering-Correlation-Id: 1fd01c85-041c-449e-f3c2-08d35ebca8aa X-Microsoft-Exchange-Diagnostics: 1;BN1PR04MB857;2:3QO4dVoJDfNxLu5Y6YV0+u8ZZUHmUa3RZbK63wmrrrD3GGwURZZrFT0Jshh9B3lzZhNKl2566FjyPXHVGeE6448+pJq87JkqVf/pxAjklgPcvDH2UqIkWD25kWqBVxZ1LvAVCvWGhQGt1Lf0HrcqCizlEK17bvTBzuMD+5VhS3TiHKn6GboaBCzXlQRJ568E;3:NmZQA0J43/yaiwNXeyC7BKsl7ZZKmMC51bLJQxBNY6VA7RunF5rKKsebbM7RTqXbN3dYsNJLu8bGtDugHidMTAMiJhZkzgeezsnG0usDq6eZGt3hvaSHvY+JPw/ACeSN;25:HAZvNRBQSy8j9M0zHY0ORCyynDGziyftEGgTCUXrhdmg3PTNlstLh+Pmw7vmFoHdXAuYPbQ3xJq4Hca58Rdm0aWBSVLHPbe9S6bXlfWnUjIQs96TLiSMMp8WdRBD9NX+2+LwIzJ2zL93ISoPZvh/RtxErTFSMj91x1sFXYhTJknSZ0/u9VtcNNohXHU0p6Mt28+GEWUqh7AmRdvGlmxoB6pMtaORUZMb+0u2r7MGZdQh65lEEqgpLUYvQu/NaDlHyI6X8TH7Y1Ok2UFfEj+1BWAEkcIBb5h7sdYiDV9qLGDRkKCuHF/BR0YCt79U9WUN63N4iqJI/hMvXLVTklvJ6A== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN1PR04MB857; X-Microsoft-Exchange-Diagnostics: 1;BN1PR04MB857;20:KK5JcVUYrUEz3KcMitFw6vviSVdvaNJS3QMj+NCEHZTHWD0BlQ82aiGqE71zUD8rsV4tXO+NjBy1vd/LXNKNgizYV8qAw5KyBnVAcdRAtBZahy7h5Nl1tyl9n9CUwV5jlcjl1Acuj2Z8cl9Ljgtzykdahm6527cDv86cQReOPXnhLLg4ASKDIA0JiY44VDBxIPPZhYchSOe9taCPPfeQ9o9Z53j7t0b/YhdK195jki15x9UNd2alClsNnruFQiyKDkOBFaeKji+TezZFyBsBHsnrQJEHElgh8aApH6ht3eWckE9lQZGVf5VnNb8/+C4DJTHP70wjThS0XsUJG89RJCUdwmTVPp4id/ts54EtaZbFrBWirQHtv9FL4hXl7h3cwFZiziY5EYglAdcmRv6XDkiEyvx2ljTt06p+KtPzsfq7eHfGsUwI9DFiJUaFnhGpFGrvCggn/Z3Mh3XXXBk+Fl+wU5oxil75uF9zEDqHa514GH8wVnZmlJncV3zhJUAMfg4tVct8/ypJ/5ioqtQ18Ejc54z1YpOs9gIJJABTa/vbup7YIg6TgNe1cfNmB153XDLA66O/WxIkxraljpazaaqs98jpZAuUMjue70x6upc= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6055026);SRVR:BN1PR04MB857;BCL:0;PCL:0;RULEID:;SRVR:BN1PR04MB857; X-Microsoft-Exchange-Diagnostics: 1;BN1PR04MB857;4:Q3M1P7/2J0ojdYWTc20CjaIZCR8vq6B49GyJSpb/U8+y7RwyEuTH/gOo9GuvPqwrq8+Q7sa469FqNk7kRwKWXeM0G5RymAmcz5UJ2xdM/1zsg8mMsd++x4iRehesnKbG/3NfHkJnyjACBVCRrBQYQJUAGTBpeno9PHzhilwVRLvJTvNdyWBGzaf/RV4Jg7YG6CztjdjNpmCOzi9iKERMaZTY7b2lpv2minudDlrRlPpMA7g1TxXkAOnCqxLdDCow5dChmGVqENVKO7dMbCxXEEnc1MjH0Ni2ttgF7SlDflvRsdczzxCksyWnyIANl45Dzyp4ylfP+V1zJ/k8SHInxD/h3z2NLlRLOA0BbEW0Q7dGB/2tkCwCtrDB2fzcicQVxRmvrKHhdfq8D59DCX/L0A== X-Forefront-PRVS: 0905A6B2C7 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(979002)(6009001)(24454002)(377454003)(77096005)(15975445007)(2950100001)(83506001)(47776003)(164054004)(5004730100002)(86362001)(42186005)(48376002)(110136002)(33646002)(1096002)(76176999)(5008740100001)(54356999)(66066001)(19580405001)(2906002)(4326007)(50986999)(6116002)(5003940100001)(3846002)(19580395003)(50466002)(81166005)(36756003)(4001350100001)(92566002)(586003)(189998001)(969003)(989001)(999001)(1009001)(1019001);DIR:OUT;SFP:1102;SCL:1;SRVR:BN1PR04MB857;H:clockwork;FPR:;SPF:None;MLV:ovrnspm;PTR:InfoNoRecords;LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;BN1PR04MB857;23:U4dABE1FZJ4Qr+Ja0F/xpicPYXjDzs3c+od3KvZoSI?= =?us-ascii?Q?0xHi9iNTBwBoyec+shTXiEnZtj2zelYZcYVuxKkbZyEqqlAYrCVKdBpvQwFm?= =?us-ascii?Q?Qf3KbWx/zQul+WI29WNAKzIpgC0i2OA6rQl7Lo1m+tAdTTXyzSrOdEQlpaQ4?= =?us-ascii?Q?zZ4S6pimgsi8hA7Edfi3zXnw4dKRdYZd5R7jsIvr1SuFV5MdevCEB+wqLG5x?= =?us-ascii?Q?L0S7ia2EWCyDxNb6OiUt4ZV2o8lWHeWVxeFRW4BJMSFWQHaDXEkpKc+4sjDj?= =?us-ascii?Q?Fd1mRiio1SAyJrkwG2jHsLRfLIeo757vhWg1tp9KuwDK/492JOMWgyGQvkD2?= =?us-ascii?Q?/Q4FZOwR2Wf4/uB4F0QIfFWMXWuVGrwus/gYc4ghQw6CdDqGMANGCd0axqYE?= =?us-ascii?Q?Fm4M+FQ0qHhhxdFufwsW+sJ3dDZ+q4T+tOFtQtENOiTGpOfZUBOENVKV3weC?= =?us-ascii?Q?nUtVgV9t4Bnf+f0gwheQgnthDjs+5wBeBemW143RtKkCNt80V/eGP0Km6PKI?= =?us-ascii?Q?Bn6s2Rc5ALWK3ImWt5kMVJYl02MQnoqzUnjHXkv9LsMdLVEzcA92Upk0KG30?= =?us-ascii?Q?AqmQeT1zmP4BVRjUNp7JT5QMs2h9Pi4ZF9HwqtZH3VOtTZDDNyZN4NnZG/II?= =?us-ascii?Q?8OIkiTeCnlSI3HkW7pOYfwDX6i3NhrMxsI7VL+MTwPs+KUgMxT8n4YogdV1W?= =?us-ascii?Q?rSDZlnw+PIDheB9u53zjwgB+G8xSnLThfxmcskATfdsBvJRVVIm31fOXMOOk?= =?us-ascii?Q?mBJG6t4bH1t2TpDWWXq7EGK9CrnMjadIBW3aRb6hovHeHD4qDhjP9NebzN4g?= =?us-ascii?Q?qmWiK52oju9M5XK603eGNbeFJ244qlRr8WLdWcJLoe2PRYWd/F2SwPKcPqCE?= =?us-ascii?Q?4QXR8el+QUHpxC5WMPlWqahpd0SGh8dJVr2pyiB2b/XjCA4UBbqf+y/TSJ9t?= =?us-ascii?Q?9oLKNG2l4mqXWmtdKTdr1VIYbrM7Zaf/k/bBP8Lc2/jK4VtJaY5oHWr4A5PU?= =?us-ascii?Q?ttt5yW7aiQpRhwycBew9KQKaVluy3fzIeAdDNRIHTb3v/KU8ikiBmo7zXUCx?= =?us-ascii?Q?5cDfeNANC+nAbRLxX+75KZp77KY+lHkRk8q8RoyIag3K91YA=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;BN1PR04MB857;5:txt+bhegvtSaFIbN8sKnwKJ3l/wByonzOVItyA1SHTM+QDH2Utt0aI/FBR0mOhkmy30Hqn/rQGn9xob+oQbr6mhhaIG/n7wiC3ARXaQqs7ZLtRjk0oSHWkyb8SJmr7UhInIkh/3Z8XsxS/hcFxkFug==;24:bNm5o3d/2Qj+gK00KXCDNBf77gQqxSuEoATIOoMb3OsokNRMfOBc3WPiXS/A6dN+08zyF8sUxjZAxMegU0+epvc0JCWlLUGw9AvVvojbbzY= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: ni.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Apr 2016 08:14:35.8822 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN1PR04MB857 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org John Stultz writes: > On Tue, Mar 15, 2016 at 11:50 AM, Gratian Crisan wrote: >> The clocksource watchdog can falsely trigger and disable the main >> clocksource when the watchdog wraps around. >> >> The reason is that an interrupt storm and/or high priority (FIFO/RR) tasks >> can preempt the timer softirq long enough for the watchdog to wrap around >> if it has a limited number of bits available by comparison to the main >> clocksource. One observed example is on a Intel Baytrail platform where TSC >> is the main clocksource, HPET is disabled due to a hardware bug and acpi_pm >> gets selected as the watchdog clocksource. >> >> Calculate the maximum number of nanoseconds the watchdog clocksource can >> represent without overflow and do not disqualify the main clocksource if >> the delta since the last time we have checked exceeds the measurement >> capabilities of the watchdog clocksource. > > Sorry for not getting back to you sooner on this. You managed to send > these both out while I was at a conference and on vacation, and so > they were deep in the mail backlog. :) No worries, I'm actually "on the road" this week too (ELC). I appreciate the reply. > So I'm sympathetic to this issue, because I remember seeing similar > problems w/ runaway SCHED_FIFO tasks w/ PREEMPT_RT. Yeah, a runaway rt thread can easily do it. That's just bad design. In our case it was a bit more subtle bc. it was a combination of high priority interrupts and rt threads that would occasionally stack up to delay the timer softirq long enough to cause the watchdog wrap. > However, its really difficult to create a solution without opening new > cases where bad clocksources will be mis-identified as good (which > your solution seems to suffer as well, measuring the time past with a > known bad clocksource can easily result in large deltas, which will be > ignored if the watchdog has a short interval). Fair point. Ultimately you have to trust one of the clocksources. I guess I was naive in thinking that the main clocksource can't drift more than what the watchdog clocksource can measure within the WATCHDOG_INTERVAL. I'm glad I don't have to deal with hardware that lobotomized. Would a simple solution that exposes the config option for the clocksource wathchdog[1] (and defaults it to on) be an acceptable alternative? It will work for us because we test the stability of the main clocksource - part of the hardware bring-up. > A previous effort on this was made here, and there's a resulting > thread that didn't come to resolution: > https://lkml.org/lkml/2015/8/17/542 Sorry I've missed it. > Way back I had tried to come up with an approach where if the time > delta was large, it was divided by the watchdog interval, and then we > just compared the remainder with the current watchdog delta to see if > they matched (closely enough). Unfortunately this didn't work out for > me then, but perhaps it deserves a second try? I've entertained that idea too but I think I was trying to optimize things too early and do everything with the mult/shift math. That first attempt failed but I do need to try harder because it would be a better general solution. > thanks > -john Thanks, -Gratian [1] >>From e942ddaba439cd6711e9eed44ceae34167b864f8 Mon Sep 17 00:00:00 2001 From: Gratian Crisan Date: Wed, 6 Apr 2016 21:20:15 -0700 Subject: [PATCH] time: Make the clocksource watchdog user configurable The clocksource watchdog is used to detect instabilities in the current clocksource. This is a beneficial feature on new/unknown hardware however it can create problems by falsely triggering when the watchdog wraps. The reason is that an interrupt storm and/or high priority (FIFO/RR) tasks can preempt the timer softirq long enough for the watchdog to wrap if it has a limited number of bits available by comparison with the main clocksource. One observed example is on a Intel Baytrail platform where TSC is the main clocksource, HPET is disabled due to a hardware bug and acpi_pm gets selected as the watchdog clocksource. Provide the option to disable the clocksource watchdog for hardware where the clocksource stability has been validated. Signed-off-by: Gratian Crisan --- arch/x86/Kconfig | 2 +- kernel/time/Kconfig | 12 +++++++++++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 2dc18605..6da5d9e 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -54,7 +54,7 @@ config X86 select CLKEVT_I8253 select CLKSRC_I8253 if X86_32 select CLOCKSOURCE_VALIDATE_LAST_CYCLE - select CLOCKSOURCE_WATCHDOG + select HAVE_CLOCKSOURCE_WATCHDOG select CLONE_BACKWARDS if X86_32 select COMPAT_OLD_SIGACTION if IA32_EMULATION select DCACHE_WORD_ACCESS diff --git a/kernel/time/Kconfig b/kernel/time/Kconfig index 4008d9f..6707f1d 100644 --- a/kernel/time/Kconfig +++ b/kernel/time/Kconfig @@ -5,7 +5,7 @@ # Options selectable by arch Kconfig # Watchdog function for clocksources to detect instabilities -config CLOCKSOURCE_WATCHDOG +config HAVE_CLOCKSOURCE_WATCHDOG bool # Architecture has extra clocksource data @@ -193,5 +193,15 @@ config HIGH_RES_TIMERS hardware is not capable then this option only increases the size of the kernel image. +config CLOCKSOURCE_WATCHDOG + bool "Clocksource watchdog" + depends on HAVE_CLOCKSOURCE_WATCHDOG + default y + help + This option enables the watchdog function for clocksources. It is + used to detect instabilities in the currently selected clocksource. + + Say Y if you are unsure. + endmenu endif -- 1.9.1