Re: [PATCH 0/3 v2] nmi perf fixes

[Robert Richter <robert.richter@amd.com>]

On 10.09.10 10:46:34, Ingo Molnar wrote:
> 
> * Don Zickus <dzickus@redhat.com> wrote:
> 
> > On Fri, Sep 10, 2010 at 01:41:40PM +0200, Peter Zijlstra wrote:
> > > On Thu, 2010-09-02 at 15:07 -0400, Don Zickus wrote:
> > > > Fixes to allow unknown nmis to pass through the perf nmi handler instead
> > > > of being swallowed.  Contains patches that are already in Ingo's tree.  Added
> > > > here for completeness.  Based on ingo/tip
> > > > 
> > > > Tested on intel/amd
> > > > 
> > > > v2: patch cleanups and consolidation, no code changes
> > > > 
> > > > Don Zickus (1):
> > > >   perf, x86: Fix accidentally ack'ing a second event on intel perf
> > > >     counter
> > > > 
> > > > Peter Zijlstra (1):
> > > >   perf, x86: Fix handle_irq return values
> > > > 
> > > > Robert Richter (1):
> > > >   perf, x86: Try to handle unknown nmis with an enabled PMU
> > > > 
> > > >  arch/x86/kernel/cpu/perf_event.c       |   59 +++++++++++++++++++++++++-------
> > > >  arch/x86/kernel/cpu/perf_event_intel.c |   15 +++++---
> > > >  arch/x86/kernel/cpu/perf_event_p4.c    |    2 +-
> > > >  3 files changed, 56 insertions(+), 20 deletions(-)
> > > 
> > > Both Ingo and I are getting Dazed and confused on our AMD machines, it
> > > started before yesterday (that is, after backing out all my recent
> > > changes it still gets dazed), so I suspect this set.
> > > 
> > > I'll look at getting a trace of the thing, but if any of you has a
> > > bright idea...
> > 
> > What are you running to create the problem?  I can try and duplicate 
> > it here.
> 
> It happens easily here - just running something like:
> 
>    perf record -g ./hackbench 10

I try to reproduce it, which systems are affected?

> 
> a couple of times triggers it. Note, unlike with the earlier bug, the 
> NMIs are not permanently 'stuck' - and everything continues working. 
> Obviously the messages are nasty looking so this is a regression we need 
> to fix.

The patch below adds ratelimits.

-Robert

--

>From 1747710d684302b806b145e5acb590ab2088e5ca Mon Sep 17 00:00:00 2001
From: Robert Richter <robert.richter@amd.com>
Date: Thu, 5 Aug 2010 18:04:31 +0200
Subject: [PATCH] x86: ratelimit NMI messages

In case of a storm of unknown NMIs the cpu get stucked. This patch
adds ratelimits to avoid this.

Signed-off-by: Robert Richter <robert.richter@amd.com>
---
 arch/x86/kernel/traps.c |   16 +++++++++-------
 1 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 60788de..97a492d 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -366,15 +366,17 @@ unknown_nmi_error(unsigned char reason, struct pt_regs *regs)
 		return;
 	}
 #endif
-	printk(KERN_EMERG
-		"Uhhuh. NMI received for unknown reason %02x on CPU %d.\n",
-			reason, smp_processor_id());
-
-	printk(KERN_EMERG "Do you have a strange power saving mode enabled?\n");
+	if (printk_ratelimit()) {
+		printk(KERN_EMERG "Uhhuh. NMI received for unknown reason"
+		       " %02x on CPU %d.\n", reason, smp_processor_id());
+		printk(KERN_EMERG
+		       "Do you have a strange power saving mode enabled?\n");
+	}
 	if (panic_on_unrecovered_nmi)
 		panic("NMI: Not continuing");
-
-	printk(KERN_EMERG "Dazed and confused, but trying to continue\n");
+	if (printk_ratelimit())
+		printk(KERN_EMERG
+		       "Dazed and confused, but trying to continue\n");
 }
 
 static notrace __kprobes void default_do_nmi(struct pt_regs *regs)
-- 
1.7.1.1



-- 
Advanced Micro Devices, Inc.
Operating System Research Center