[PATCH v2] Add hard/soft lockup debugger entry points

[PATCH v2] Add hard/soft lockup debugger entry points

[Jeff Merkey]

This patch adds an export which can be set by system debuggers to direct
the hard lockup and soft lockup detector to trigger a breakpoint exception
and enter a debugger if one is active.  It is assumed that if someone
sets this variable, then an breakpoint handler of some sort will be actively
loaded or registered via the notify die handler chain.

This addition is extremely useful for debugging hard and soft lockups
real time and quickly from a console debugger.

Signed-off-by: Jeff Merkey <linux.mdb@gmail.com>
---
 kernel/watchdog.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index b3ace6e..c28e58c 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -23,6 +23,7 @@
 #include <linux/workqueue.h>
 
 #include <asm/irq_regs.h>
+#include <linux/kgdb.h>
 #include <linux/kvm_para.h>
 #include <linux/perf_event.h>
 #include <linux/kthread.h>
@@ -108,6 +109,9 @@ static DEFINE_PER_CPU(struct perf_event *, watchdog_ev);
 #endif
 static unsigned long soft_lockup_nmi_warn;
 
+int debug_watchdog_lockups;
+EXPORT_SYMBOL_GPL(debug_watchdog_lockups);
+
 /* boot commands */
 /*
  * Should we panic when a soft-lockup or hard-lockup occurs:
@@ -358,6 +362,9 @@ static void watchdog_overflow_callback(struct perf_event *event,
 		else
 			dump_stack();
 
+		if (debug_watchdog_lockups)
+			arch_kgdb_breakpoint();
+
 		/*
 		 * Perform all-CPU dump only once to avoid multiple hardlockups
 		 * generating interleaving traces
@@ -478,6 +485,9 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer)
 		else
 			dump_stack();
 
+		if (debug_watchdog_lockups)
+			arch_kgdb_breakpoint();
+
 		if (softlockup_all_cpu_backtrace) {
 			/* Avoid generating two back traces for current
 			 * given that one is already made above
-- 
1.8.3.1

Re: [PATCH v2] Add hard/soft lockup debugger entry points

[Chris Metcalf]

On 01/26/2016 12:04 PM, Jeff Merkey wrote:
> This patch adds an export which can be set by system debuggers to direct
> the hard lockup and soft lockup detector to trigger a breakpoint exception
> and enter a debugger if one is active.  It is assumed that if someone
> sets this variable, then an breakpoint handler of some sort will be actively
> loaded or registered via the notify die handler chain.
>
> This addition is extremely useful for debugging hard and soft lockups
> real time and quickly from a console debugger.
>
> Signed-off-by: Jeff Merkey<linux.mdb@gmail.com>
> ---
>   kernel/watchdog.c | 10 ++++++++++
>   1 file changed, 10 insertions(+)

You probably should reach out to someone who uses this stuff more
regularly - I actually wonder if the kgdb_breakpoint() API is the
right thing, though, not the internal arch_kgdb_breakpoint().

Of course any of these strategies also assume you are building
the kernel with CONFIG_KGDB set, and I'm pretty sure will cause
your build to fail if it isn't.  You likely need to guard this
stuff locally within watchdog.c for !CONFIG_KGDB.

-- 
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com

Re: [PATCH v2] Add hard/soft lockup debugger entry points

[Jeff Merkey]

On 1/26/16, Chris Metcalf <cmetcalf@ezchip.com> wrote:
> On 01/26/2016 12:04 PM, Jeff Merkey wrote:
>> This patch adds an export which can be set by system debuggers to direct
>> the hard lockup and soft lockup detector to trigger a breakpoint
>> exception
>> and enter a debugger if one is active.  It is assumed that if someone
>> sets this variable, then an breakpoint handler of some sort will be
>> actively
>> loaded or registered via the notify die handler chain.
>>
>> This addition is extremely useful for debugging hard and soft lockups
>> real time and quickly from a console debugger.
>>
>> Signed-off-by: Jeff Merkey<linux.mdb@gmail.com>
>> ---
>>   kernel/watchdog.c | 10 ++++++++++
>>   1 file changed, 10 insertions(+)
>
> You probably should reach out to someone who uses this stuff more
> regularly - I actually wonder if the kgdb_breakpoint() API is the
> right thing, though, not the internal arch_kgdb_breakpoint().
>
> Of course any of these strategies also assume you are building
> the kernel with CONFIG_KGDB set, and I'm pretty sure will cause
> your build to fail if it isn't.  You likely need to guard this
> stuff locally within watchdog.c for !CONFIG_KGDB.
>
> --
> Chris Metcalf, EZChip Semiconductor
> http://www.ezchip.com
>
>

I'll go review that code carefully.  What would ideal is to have the
breakpoint abstraction independent of kgdb entirely.  That would not
be a simple patch but more involved.  We really just need the
breakpoint instruction for each arch, not all of kgdb and it should
not depend on KGB being enabled.  I looked over those includes and
this may work, but you are correct, it's not squeaky clean to do it
that way.  I'll get to work on v3.

:-)

Jeff

Re: [PATCH v2] Add hard/soft lockup debugger entry points

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 1409 bytes --]

Hi Jeff,

[auto build test ERROR on v4.5-rc1]
[also build test ERROR on next-20160125]
[if your patch is applied to the wrong git tree, please drop us a note to help improving the system]

url:    https://github.com/0day-ci/linux/commits/Jeff-Merkey/Add-hard-soft-lockup-debugger-entry-points/20160127-010930
config: xtensa-allyesconfig (attached as .config)
reproduce:
        wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=xtensa 

All errors (new ones prefixed by >>):

   kernel/watchdog.c: In function 'watchdog_timer_fn':
>> kernel/watchdog.c:489:4: error: implicit declaration of function 'arch_kgdb_breakpoint' [-Werror=implicit-function-declaration]
       arch_kgdb_breakpoint();
       ^
   cc1: some warnings being treated as errors

vim +/arch_kgdb_breakpoint +489 kernel/watchdog.c

   483			if (regs)
   484				show_regs(regs);
   485			else
   486				dump_stack();
   487	
   488			if (debug_watchdog_lockups)
 > 489				arch_kgdb_breakpoint();
   490	
   491			if (softlockup_all_cpu_backtrace) {
   492				/* Avoid generating two back traces for current

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 44066 bytes --]

Re: [PATCH v2] Add hard/soft lockup debugger entry points

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

Hi Jeff,

[auto build test ERROR on v4.5-rc1]
[also build test ERROR on next-20160125]
[if your patch is applied to the wrong git tree, please drop us a note to help improving the system]

url:    https://github.com/0day-ci/linux/commits/Jeff-Merkey/Add-hard-soft-lockup-debugger-entry-points/20160127-010930
config: sparc64-defconfig (attached as .config)
reproduce:
        wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=sparc64 

All errors (new ones prefixed by >>):

   kernel/built-in.o: In function `watchdog_timer_fn':
>> watchdog.c:(.text+0x736ec): undefined reference to `arch_kgdb_breakpoint'

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 16661 bytes --]

Re: [PATCH v2] Add hard/soft lockup debugger entry points

[Jeff Merkey]

On 1/26/16, kbuild test robot <lkp@intel.com> wrote:
> Hi Jeff,
>
> [auto build test ERROR on v4.5-rc1]
> [also build test ERROR on next-20160125]
> [if your patch is applied to the wrong git tree, please drop us a note to
> help improving the system]
>
> url:
> https://github.com/0day-ci/linux/commits/Jeff-Merkey/Add-hard-soft-lockup-debugger-entry-points/20160127-010930
> config: sparc64-defconfig (attached as .config)
> reproduce:
>         wget
> https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross
> -O ~/bin/make.cross
>         chmod +x ~/bin/make.cross
>         # save the attached .config to linux build tree
>         make.cross ARCH=sparc64
>
> All errors (new ones prefixed by >>):
>
>    kernel/built-in.o: In function `watchdog_timer_fn':
>>> watchdog.c:(.text+0x736ec): undefined reference to
>>> `arch_kgdb_breakpoint'
>
> ---
> 0-DAY kernel test infrastructure                Open Source Technology
> Center
> https://lists.01.org/pipermail/kbuild-all                   Intel
> Corporation
>

I love this test robot.  What an incredibly useful tool.  Well, looks
like a I quite a bit of work to do.

Jeff