[PATCH v3] watchdog: touch_nmi_watchdog should only touch local cpu not every one

[PATCH v3] watchdog: touch_nmi_watchdog should only touch local cpu not every one

[Don Zickus]

I ran into a scenario where while one cpu was stuck and should have panic'd
because of the NMI watchdog, it didn't.  The reason was another cpu was spewing
stack dumps on to the console.  Upon investigation, I noticed that when writing
to the console and also when dumping the stack, the watchdog is touched.

This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu
just spins forever.

This change causes the semantics of touch_nmi_watchdog to be changed slightly.
Previously, I accidentally changed the semantics and we noticed there was a
codepath in which touch_nmi_watchdog could be touched from a preemtible area.
That caused a BUG() to happen when CONFIG_DEBUG_PREEMPT was enabled.  I believe
it was the acpi code.

My attempt here re-introduces the change to have the touch_nmi_watchdog() code
only touch the local cpu instead of all of the cpus.  But instead of using
__get_cpu_var(), I use the __raw_get_cpu_var() version.

This avoids the preemption problem.  However my reasoning wasn't because I was
trying to be lazy.  Instead I rationalized it as, well if preemption is enabled
then interrupts should be enabled to and the NMI watchdog will have no reason
to trigger.  So it won't matter if the wrong cpu is touched because the percpu
interrupt counters the NMI watchdog uses should still be incrementing.

V3: Really remove touch_all_nmi_watchdogs()
V2: Remove touch_all_nmi_watchdogs()

Signed-off-by: Don Zickus <dzickus@redhat.com>
---
 kernel/watchdog.c |   17 +++++++++--------
 1 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index dc8e168..09fddd7 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -141,14 +141,15 @@ void touch_all_softlockup_watchdogs(void)
 #ifdef CONFIG_HARDLOCKUP_DETECTOR
 void touch_nmi_watchdog(void)
 {
-	if (watchdog_enabled) {
-		unsigned cpu;
+	/*
+	 * Using __raw here because some code paths have
+	 * preemption enabled.  If preemption is enabled
+	 * then interrupts should be enabled too, in which
+	 * case we shouldn't have to worry about the watchdog
+	 * going off.
+	 */
+	__raw_get_cpu_var(watchdog_nmi_touch) = true;
 
-		for_each_present_cpu(cpu) {
-			if (per_cpu(watchdog_nmi_touch, cpu) != true)
-				per_cpu(watchdog_nmi_touch, cpu) = true;
-		}
-	}
 	touch_softlockup_watchdog();
 }
 EXPORT_SYMBOL(touch_nmi_watchdog);
-- 
1.7.2.3

Re: [PATCH v3] watchdog: touch_nmi_watchdog should only touch local cpu not every one

[Ingo Molnar]

* Don Zickus <dzickus@redhat.com> wrote:

> I ran into a scenario where while one cpu was stuck and should have panic'd 
> because of the NMI watchdog, it didn't.  The reason was another cpu was spewing 
> stack dumps on to the console.  Upon investigation, I noticed that when writing to 
> the console and also when dumping the stack, the watchdog is touched.
> 
> This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu 
> just spins forever.

Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all 
the other CPUs a lot more noisy - which might only 'lock up' because this CPU is 
stuck spewing oopses, right?

Andrew, what would be your preference?

Thanks,

	Ingo

Re: [PATCH v3] watchdog: touch_nmi_watchdog should only touch local cpu not every one

[Andrew Morton]

On Wed, 10 Nov 2010 08:49:41 +0100 Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Don Zickus <dzickus@redhat.com> wrote:
> 
> > I ran into a scenario where while one cpu was stuck and should have panic'd 
> > because of the NMI watchdog, it didn't.  The reason was another cpu was spewing 
> > stack dumps on to the console.  Upon investigation, I noticed that when writing to 
> > the console and also when dumping the stack, the watchdog is touched.
> > 
> > This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu 
> > just spins forever.
> 
> Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all 
> the other CPUs a lot more noisy - which might only 'lock up' because this CPU is 
> stuck spewing oopses, right?
> 
> Andrew, what would be your preference?
> 

I saw you were cc'ed and felt rather pleased that I wouldn't have to
decide ;)

I guess the NMI really is a cpu-local concept rather than a
machine-wide one.  And touch_nmi_watchdog() says "this CPU isn't
stuck", rather than "this CPU and all those others aren't stuck", yes?

Re: [PATCH v3] watchdog: touch_nmi_watchdog should only touch local cpu not every one

[Don Zickus]

On Wed, Nov 10, 2010 at 08:49:41AM +0100, Ingo Molnar wrote:
> 
> * Don Zickus <dzickus@redhat.com> wrote:
> 
> > I ran into a scenario where while one cpu was stuck and should have panic'd 
> > because of the NMI watchdog, it didn't.  The reason was another cpu was spewing 
> > stack dumps on to the console.  Upon investigation, I noticed that when writing to 
> > the console and also when dumping the stack, the watchdog is touched.
> > 
> > This causes all the cpus to reset their NMI watchdog flags and the 'stuck' cpu 
> > just spins forever.
> 
> Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make all 
> the other CPUs a lot more noisy - which might only 'lock up' because this CPU is 
> stuck spewing oopses, right?

When you say the other CPUs will be a lot more noisy, is that because they
are busy processing backtraces for the first cpu to spew?  I guess I don't
understand how the other CPUs could have their interrupts off the whole
time while the first cpu is spewing a backtrace (just trying to educate
myself).

Cheers,
Don

Re: [PATCH v3] watchdog: touch_nmi_watchdog should only touch local cpu not every one

[Ingo Molnar]

* Don Zickus <dzickus@redhat.com> wrote:

> > Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make 
> > all the other CPUs a lot more noisy - which might only 'lock up' because this 
> > CPU is stuck spewing oopses, right?
> 
> When you say the other CPUs will be a lot more noisy, is that because they are 
> busy processing backtraces for the first cpu to spew?  I guess I don't understand 
> how the other CPUs could have their interrupts off the whole time while the first 
> cpu is spewing a backtrace (just trying to educate myself).

Say the kernel crashes on a CPU and keeps spewing new oopses, while write-holding 
tasklist_lock.

Any other CPU that delivers a signal from IRQ context, trying to take the 
tasklist_lock, will loop indefinitely until that crashing CPU releases the lock.

In that case the 'secondary' NMI warnings from all other CPUs (eventually every CPU 
gets stuck in such a scenario) will start spewing NMI lockup messages.

Dunno. Maybe we should do your change - but also have an option to 'shut up' the 
kernel after the first hard oops [not warning]. That would silence the secondary NMI 
watchdog messages as well.

Thanks,

	Ingo

Re: [PATCH v3] watchdog: touch_nmi_watchdog should only touch local cpu not every one

[Don Zickus]

On Wed, Nov 10, 2010 at 07:07:11PM +0100, Ingo Molnar wrote:
> 
> * Don Zickus <dzickus@redhat.com> wrote:
> 
> > > Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make 
> > > all the other CPUs a lot more noisy - which might only 'lock up' because this 
> > > CPU is stuck spewing oopses, right?
> > 
> > When you say the other CPUs will be a lot more noisy, is that because they are 
> > busy processing backtraces for the first cpu to spew?  I guess I don't understand 
> > how the other CPUs could have their interrupts off the whole time while the first 
> > cpu is spewing a backtrace (just trying to educate myself).
> 
> Say the kernel crashes on a CPU and keeps spewing new oopses, while write-holding 
> tasklist_lock.
> 
> Any other CPU that delivers a signal from IRQ context, trying to take the 
> tasklist_lock, will loop indefinitely until that crashing CPU releases the lock.
> 
> In that case the 'secondary' NMI warnings from all other CPUs (eventually every CPU 
> gets stuck in such a scenario) will start spewing NMI lockup messages.
> 
> Dunno. Maybe we should do your change - but also have an option to 'shut up' the 
> kernel after the first hard oops [not warning]. That would silence the secondary NMI 
> watchdog messages as well.

You mean inside the panic() routine? like a ratelimit?

Cheers,
Don

Re: [PATCH v3] watchdog: touch_nmi_watchdog should only touch local cpu not every one

[Ingo Molnar]

* Don Zickus <dzickus@redhat.com> wrote:

> On Wed, Nov 10, 2010 at 07:07:11PM +0100, Ingo Molnar wrote:
> > 
> > * Don Zickus <dzickus@redhat.com> wrote:
> > 
> > > > Hm, the flip side is that if a CPU is stuck spewing backtraces, we will now make 
> > > > all the other CPUs a lot more noisy - which might only 'lock up' because this 
> > > > CPU is stuck spewing oopses, right?
> > > 
> > > When you say the other CPUs will be a lot more noisy, is that because they are 
> > > busy processing backtraces for the first cpu to spew?  I guess I don't understand 
> > > how the other CPUs could have their interrupts off the whole time while the first 
> > > cpu is spewing a backtrace (just trying to educate myself).
> > 
> > Say the kernel crashes on a CPU and keeps spewing new oopses, while write-holding 
> > tasklist_lock.
> > 
> > Any other CPU that delivers a signal from IRQ context, trying to take the 
> > tasklist_lock, will loop indefinitely until that crashing CPU releases the lock.
> > 
> > In that case the 'secondary' NMI warnings from all other CPUs (eventually every CPU 
> > gets stuck in such a scenario) will start spewing NMI lockup messages.
> > 
> > Dunno. Maybe we should do your change - but also have an option to 'shut up' the 
> > kernel after the first hard oops [not warning]. That would silence the secondary NMI 
> > watchdog messages as well.
> 
> You mean inside the panic() routine? like a ratelimit?

A ratelimit, but some really serious one - like only one crash displayed per 10 
minutes, or so - to give the user time to make a picture of the first crash, if it's 
still visible on the screen or so.

Thanks,

	Ingo

Re: [PATCH v3] watchdog: touch_nmi_watchdog should only touch local cpu not every one

[Don Zickus]

On Wed, Nov 10, 2010 at 08:14:32PM +0100, Ingo Molnar wrote:
> > > Dunno. Maybe we should do your change - but also have an option to 'shut up' the 
> > > kernel after the first hard oops [not warning]. That would silence the secondary NMI 
> > > watchdog messages as well.
> > 
> > You mean inside the panic() routine? like a ratelimit?
> 
> A ratelimit, but some really serious one - like only one crash displayed per 10 
> minutes, or so - to give the user time to make a picture of the first crash, if it's 
> still visible on the screen or so.

Well I hacked up something.  It seems to work, but I am not sure I put the
check in the right spot.

Comments welcome.

-----------8<---cut---8<-----

From: Don Zickus <dzickus@redhat.com>
Date: Mon, 15 Nov 2010 13:09:31 -0500
Subject: [PATCH] panic:  ratelimit panic messages

Sometimes when things go bad, so much spew is coming on the console it is hard
to figure out what happened.  This patch allows you to ratelimit the panic
messages with the intent that the first panic message will provide the info
we need to figure out what happened.

Adds new kernel param 'panic_ratelimit=on/<integer in seconds>'

Signed-off-by: Don Zickus <dzickus@redhat.com>
---
 Documentation/kernel-parameters.txt |    6 ++++++
 kernel/panic.c                      |   30 ++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+), 0 deletions(-)

diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index ed45e98..5d69064 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -1815,6 +1815,12 @@ and is between 256 and 4096 characters. It is defined in the file
 	panic=		[KNL] Kernel behaviour on panic
 			Format: <timeout>
 
+	panic_ratelimit=	[KNL] ratelimit the panic messages
+			Useful for slowing down multiple panics to capture
+			the first one before it scrolls off the screen
+			Format: "on" or some integer in seconds
+			"on" defaults to 10 minutes
+
 	parkbd.port=	[HW] Parallel port number the keyboard adapter is
 			connected to, default is 0.
 			Format: <parport#>
diff --git a/kernel/panic.c b/kernel/panic.c
index 4c13b1a..fe89e04 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -23,6 +23,7 @@
 #include <linux/init.h>
 #include <linux/nmi.h>
 #include <linux/dmi.h>
+#include <linux/ratelimit.h>
 
 #define PANIC_TIMER_STEP 100
 #define PANIC_BLINK_SPD 18
@@ -48,6 +49,31 @@ static long no_blink(int state)
 long (*panic_blink)(int state);
 EXPORT_SYMBOL(panic_blink);
 
+/* setting default to 0 effectively disables it */
+DEFINE_RATELIMIT_STATE(panic_ratelimit_state, 0, 1);
+
+static int __init panic_ratelimit_setup(char *str)
+{
+	int interval;
+
+	if (!strncmp(str, "on", 2))
+		/* default to 10 minutes */
+		interval = 600 * HZ;
+	else
+		interval = simple_strtoul(str, NULL, 0) * HZ;
+
+	panic_ratelimit_state.interval = interval;
+	return 1;
+}
+__setup("panic_ratelimit=", panic_ratelimit_setup);
+
+static int __panic_ratelimit(const char *func)
+{
+	return ___ratelimit(&panic_ratelimit_state, func);
+}
+
+#define panic_ratelimit() __panic_ratelimit(__func__)
+
 /**
  *	panic - halt the system
  *	@fmt: The text string to print
@@ -63,6 +89,10 @@ NORET_TYPE void panic(const char * fmt, ...)
 	long i, i_next = 0;
 	int state = 0;
 
+	if (!panic_ratelimit())
+		for(;;)
+			cpu_relax();
+
 	/*
 	 * It's possible to come here directly from a panic-assertion and
 	 * not have preempt disabled. Some functions called from here want
-- 
1.7.2.3