From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933510AbcATCHn (ORCPT ); Tue, 19 Jan 2016 21:07:43 -0500 Received: from mail-oi0-f46.google.com ([209.85.218.46]:32870 "EHLO mail-oi0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751549AbcATCHg (ORCPT ); Tue, 19 Jan 2016 21:07:36 -0500 Date: Tue, 19 Jan 2016 19:07:32 -0700 From: Jeff Merkey To: linux-kernel@vger.kernel.org Cc: John Stultz , Thomas Gleixner Subject: [PATCH] Fix Hard Lockup in ktime_get_ts64 with debuggers Message-ID: <20160120020732.GA16231@localhost.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This patch fixes a hard system lockup that occurs when a system has been held by a kernel debugger such as kgdb, kdb, or mdb on a multiprocessor system with all processors suspended then released after 20-30 minutes of debugging. SUMMARY An SMP multiprocessor capable kernel debugger must stop all processors on a target system and trigger them to be held inside a NMI polling routine while tracing and debugging active events on a target system. This is typically accomplished by triggering an NMI interrupt via an IPI call to each processor then directing the processors to enter a suspend code loop from the NMI handlers. While the processors are held suspended, the system hardware timer will continue to run as well as other hardware components. Upon exiting the debugger console, the processors are released and the system is expected to resume normal operations. In the case that causes this bug, the Linux timekeeper subsystem sees a very large delta value for the time in nanoseconds elapsed then attempts to process this large number. Since the function macro timespec_add_ns calls an inline macro function that manually subtracts the values longhand rather than use the processor to perform the divide operation, this results in the hard lockup detector causing a panic because the function is looping and hung for about 4 minutes on average calling this longhand divide macro __iter_div_u64_rem. CONDITIONS THAT RESULT IN THIS LOCKUP Holding the system processors suspended in the NMI handlers for 20 minutes to several hours while examining and debugging a target system will trigger this bug upon exit from the debugger console on either kgdb, kdb, or mdb. It appears to be caused by the delta shift of the counters in the hardware clock in ns being calculated and passed to the function to update the timespec. The function being replaced is little more than a cycle hog on the system and needs to be replaced witb someone that does not trigger the hard lockup detector if it sees a really big number for nanoseconds. Signed-off-by: Jeff Merkey --- include/linux/math64.h | 17 +++++++++++++++++ include/linux/time.h | 2 +- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/include/linux/math64.h b/include/linux/math64.h index 6e8b5b2..6769a3d 100644 --- a/include/linux/math64.h +++ b/include/linux/math64.h @@ -22,6 +22,15 @@ static inline u64 div_u64_rem(u64 dividend, u32 divisor, u32 *remainder) } /** + * div_s64_rem64 - unsigned 64bit divide with 32bit divisor with 64bit remainder + */ +static inline u64 div_u64_rem64(u64 dividend, u32 divisor, u64 *remainder) +{ + *remainder = dividend % divisor; + return dividend / divisor; +} + +/** * div_s64_rem - signed 64bit divide with 32bit divisor with remainder */ static inline s64 div_s64_rem(s64 dividend, s32 divisor, s32 *remainder) @@ -68,6 +77,14 @@ static inline u64 div_u64_rem(u64 dividend, u32 divisor, u32 *remainder) } #endif +#ifndef div_u64_rem64 +static inline u64 div_u64_rem64(u64 dividend, u32 divisor, u64 *remainder) +{ + *remainder = do_div(dividend, divisor); + return dividend; +} +#endif + #ifndef div_s64_rem extern s64 div_s64_rem(s64 dividend, s32 divisor, s32 *remainder); #endif diff --git a/include/linux/time.h b/include/linux/time.h index 297f09f..e9e1fbc 100644 --- a/include/linux/time.h +++ b/include/linux/time.h @@ -258,7 +258,7 @@ extern struct timeval ns_to_timeval(const s64 nsec); */ static __always_inline void timespec_add_ns(struct timespec *a, u64 ns) { - a->tv_sec += __iter_div_u64_rem(a->tv_nsec + ns, NSEC_PER_SEC, &ns); + a->tv_sec += div_u64_rem64(a->tv_nsec + ns, NSEC_PER_SEC, &ns); a->tv_nsec = ns; } -- 1.8.3.1