[PATCH 0/3 v2] nmi perf fixes

[PATCH 0/3 v2] nmi perf fixes

[Don Zickus]

Fixes to allow unknown nmis to pass through the perf nmi handler instead
of being swallowed.  Contains patches that are already in Ingo's tree.  Added
here for completeness.  Based on ingo/tip

Tested on intel/amd

v2: patch cleanups and consolidation, no code changes

Don Zickus (1):
  perf, x86: Fix accidentally ack'ing a second event on intel perf
    counter

Peter Zijlstra (1):
  perf, x86: Fix handle_irq return values

Robert Richter (1):
  perf, x86: Try to handle unknown nmis with an enabled PMU

 arch/x86/kernel/cpu/perf_event.c       |   59 +++++++++++++++++++++++++-------
 arch/x86/kernel/cpu/perf_event_intel.c |   15 +++++---
 arch/x86/kernel/cpu/perf_event_p4.c    |    2 +-
 3 files changed, 56 insertions(+), 20 deletions(-)

-- 
1.7.2.2

[PATCH 1/3] perf, x86: Fix accidentally ack'ing a second event on intel perf counter

[Don Zickus]

During testing of a patch to stop having the perf subsytem swallow nmis,
it was uncovered that Nehalem boxes were randomly getting unknown nmis
when using the perf tool.

Moving the ack'ing of the PMI closer to when we get the status allows
the hardware to properly re-set the PMU bit signaling another PMI was
triggered during the processing of the first PMI.  This allows the new
logic for dealing with the shortcomings of multiple PMIs to handle the
extra NMI by 'eat'ing it later.

Now one can wonder why are we getting a second PMI when we disable all
the PMUs in the begining of the NMI handler to prevent such a case, for
that I do not know.  But I know the fix below helps deal with this quirk.

Tested on multiple Nehalems where the problem was occuring.  With the
patch, the code now loops a second time to handle the second PMI (whereas
before it was not).

Signed-off-by: Don Zickus <dzickus@redhat.com>
---
 arch/x86/kernel/cpu/perf_event_intel.c |    6 ++----
 1 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
index d8d86d0..1297bf1 100644
--- a/arch/x86/kernel/cpu/perf_event_intel.c
+++ b/arch/x86/kernel/cpu/perf_event_intel.c
@@ -712,7 +712,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	struct perf_sample_data data;
 	struct cpu_hw_events *cpuc;
 	int bit, loops;
-	u64 ack, status;
+	u64 status;
 
 	perf_sample_data_init(&data, 0);
 
@@ -728,6 +728,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 
 	loops = 0;
 again:
+	intel_pmu_ack_status(status);
 	if (++loops > 100) {
 		WARN_ONCE(1, "perfevents: irq loop stuck!\n");
 		perf_event_print_debug();
@@ -736,7 +737,6 @@ again:
 	}
 
 	inc_irq_stat(apic_perf_irqs);
-	ack = status;
 
 	intel_pmu_lbr_read();
 
@@ -761,8 +761,6 @@ again:
 			x86_pmu_stop(event);
 	}
 
-	intel_pmu_ack_status(ack);
-
 	/*
 	 * Repeat if there is more work to be done:
 	 */
-- 
1.7.2.2

Re: [PATCH 1/3] perf, x86: Fix accidentally ack'ing a second event on intel perf counter

[Cyrill Gorcunov]

On Thu, Sep 02, 2010 at 03:07:47PM -0400, Don Zickus wrote:
> During testing of a patch to stop having the perf subsytem swallow nmis,
> it was uncovered that Nehalem boxes were randomly getting unknown nmis
> when using the perf tool.
> 
> Moving the ack'ing of the PMI closer to when we get the status allows
> the hardware to properly re-set the PMU bit signaling another PMI was
> triggered during the processing of the first PMI.  This allows the new
> logic for dealing with the shortcomings of multiple PMIs to handle the
> extra NMI by 'eat'ing it later.
> 
> Now one can wonder why are we getting a second PMI when we disable all
> the PMUs in the begining of the NMI handler to prevent such a case, for
> that I do not know.  But I know the fix below helps deal with this quirk.
> 
> Tested on multiple Nehalems where the problem was occuring.  With the
> patch, the code now loops a second time to handle the second PMI (whereas
> before it was not).
> 
> Signed-off-by: Don Zickus <dzickus@redhat.com>
> ---
>  arch/x86/kernel/cpu/perf_event_intel.c |    6 ++----
>  1 files changed, 2 insertions(+), 4 deletions(-)
> 

Hi Don,

I might be missing something (I'm sure I'm actually) so enlighten me
a bit please

> diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
> index d8d86d0..1297bf1 100644
> --- a/arch/x86/kernel/cpu/perf_event_intel.c
> +++ b/arch/x86/kernel/cpu/perf_event_intel.c
> @@ -712,7 +712,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
>  	struct perf_sample_data data;
>  	struct cpu_hw_events *cpuc;
>  	int bit, loops;
> -	u64 ack, status;
> +	u64 status;
>  

Lets assume 1 counters is triggered and global bit is set as well

we have here

	status = intel_pmu_get_status();

>  	perf_sample_data_init(&data, 0);
>  
> @@ -728,6 +728,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
>  
>  	loops = 0;
>  again:
> +	intel_pmu_ack_status(status);

So here we write just being read value back to CTRL register and _if_ new
overflow happened in this window we've cleared it without processing.

>  	if (++loops > 100) {
>  		WARN_ONCE(1, "perfevents: irq loop stuck!\n");
>  		perf_event_print_debug();
> @@ -736,7 +737,6 @@ again:
>  	}
>  
>  	inc_irq_stat(apic_perf_irqs);
> -	ack = status;
>  
>  	intel_pmu_lbr_read();
>  
> @@ -761,8 +761,6 @@ again:
>  			x86_pmu_stop(event);
>  	}
>  
> -	intel_pmu_ack_status(ack);
> -

Here we cleared bits in "status" variable and then we read
status register again without cleaning bits in real physical
register which confuses me.

>  	/*
>  	 * Repeat if there is more work to be done:
>  	 */
> -- 
> 1.7.2.2
> 

	-- Cyrill

Re: [PATCH 1/3] perf, x86: Fix accidentally ack'ing a second event on intel perf counter

[Don Zickus]

On Thu, Sep 02, 2010 at 11:26:27PM +0400, Cyrill Gorcunov wrote:
> Lets assume 1 counters is triggered and global bit is set as well
> 
> we have here
> 
> 	status = intel_pmu_get_status();
> 
> >  	perf_sample_data_init(&data, 0);
> >  
> > @@ -728,6 +728,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
> >  
> >  	loops = 0;
> >  again:
> > +	intel_pmu_ack_status(status);
> 
> So here we write just being read value back to CTRL register and _if_ new
> overflow happened in this window we've cleared it without processing.

No, you have to write a '1' to clear.  If a new bit is set after we read
it, then our 'status' var should have a '0' in that bit and thus will not
get cleared when we ack it.

Cheers,
Don

Re: [PATCH 1/3] perf, x86: Fix accidentally ack'ing a second event on intel perf counter

[Cyrill Gorcunov]

On Thu, Sep 02, 2010 at 04:00:21PM -0400, Don Zickus wrote:
> On Thu, Sep 02, 2010 at 11:26:27PM +0400, Cyrill Gorcunov wrote:
> > Lets assume 1 counters is triggered and global bit is set as well
> > 
> > we have here
> > 
> > 	status = intel_pmu_get_status();
> > 
> > >  	perf_sample_data_init(&data, 0);
> > >  
> > > @@ -728,6 +728,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
> > >  
> > >  	loops = 0;
> > >  again:
> > > +	intel_pmu_ack_status(status);
> > 
> > So here we write just being read value back to CTRL register and _if_ new
> > overflow happened in this window we've cleared it without processing.
> 
> No, you have to write a '1' to clear.  If a new bit is set after we read
> it, then our 'status' var should have a '0' in that bit and thus will not
> get cleared when we ack it.
> 

Yeah, messed {0;1} values, thanks!

> Cheers,
> Don
> 
	-- Cyrill

[tip:perf/urgent] perf, x86: Fix accidentally ack'ing a second event on intel perf counter

[tip-bot for Don Zickus]

Commit-ID:  2e556b5b320838fde98480a1f6cf220a5af200fc
Gitweb:     http://git.kernel.org/tip/2e556b5b320838fde98480a1f6cf220a5af200fc
Author:     Don Zickus <dzickus@redhat.com>
AuthorDate: Thu, 2 Sep 2010 15:07:47 -0400
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 3 Sep 2010 08:05:17 +0200

perf, x86: Fix accidentally ack'ing a second event on intel perf counter

During testing of a patch to stop having the perf subsytem
swallow nmis, it was uncovered that Nehalem boxes were randomly
getting unknown nmis when using the perf tool.

Moving the ack'ing of the PMI closer to when we get the status
allows the hardware to properly re-set the PMU bit signaling
another PMI was triggered during the processing of the first
PMI.  This allows the new logic for dealing with the
shortcomings of multiple PMIs to handle the extra NMI by
'eat'ing it later.

Now one can wonder why are we getting a second PMI when we
disable all the PMUs in the begining of the NMI handler to
prevent such a case, for that I do not know.  But I know the fix
below helps deal with this quirk.

Tested on multiple Nehalems where the problem was occuring.
With the patch, the code now loops a second time to handle the
second PMI (whereas before it was not).

Signed-off-by: Don Zickus <dzickus@redhat.com>
Cc: peterz@infradead.org
Cc: robert.richter@amd.com
Cc: gorcunov@gmail.com
Cc: fweisbec@gmail.com
Cc: ying.huang@intel.com
Cc: ming.m.lin@intel.com
Cc: eranian@google.com
LKML-Reference: <1283454469-1909-2-git-send-email-dzickus@redhat.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 arch/x86/kernel/cpu/perf_event_intel.c |    6 ++----
 1 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
index d8d86d0..1297bf1 100644
--- a/arch/x86/kernel/cpu/perf_event_intel.c
+++ b/arch/x86/kernel/cpu/perf_event_intel.c
@@ -712,7 +712,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	struct perf_sample_data data;
 	struct cpu_hw_events *cpuc;
 	int bit, loops;
-	u64 ack, status;
+	u64 status;
 
 	perf_sample_data_init(&data, 0);
 
@@ -728,6 +728,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 
 	loops = 0;
 again:
+	intel_pmu_ack_status(status);
 	if (++loops > 100) {
 		WARN_ONCE(1, "perfevents: irq loop stuck!\n");
 		perf_event_print_debug();
@@ -736,7 +737,6 @@ again:
 	}
 
 	inc_irq_stat(apic_perf_irqs);
-	ack = status;
 
 	intel_pmu_lbr_read();
 
@@ -761,8 +761,6 @@ again:
 			x86_pmu_stop(event);
 	}
 
-	intel_pmu_ack_status(ack);
-
 	/*
 	 * Repeat if there is more work to be done:
 	 */

Re: [tip:perf/urgent] perf, x86: Fix accidentally ack'ing a second event on intel perf counter

[Yinghai Lu]

Can you put this into perf branch ?

Thanks

Yinghai

[PATCH] x86,nmi: move unknown_nmi_panic to traps.c

So we use it even LOCKUP_DETECTOR is defined.
need Robert/Peter/Don's patch...

that will keep unknown_nmi_panic to have same behavoir before hw nmi watchdog

Signed-off-by: Yinghai Lu <yinghai@kernel.org>

---
 arch/x86/include/asm/nmi.h    |    8 --------
 arch/x86/kernel/apic/hw_nmi.c |    1 -
 arch/x86/kernel/apic/nmi.c    |   27 ---------------------------
 arch/x86/kernel/traps.c       |   29 ++++++++++++++++++++++++++++-
 kernel/sysctl.c               |    4 +++-
 5 files changed, 31 insertions(+), 38 deletions(-)

Index: linux-2.6/arch/x86/kernel/apic/nmi.c
===================================================================
--- linux-2.6.orig/arch/x86/kernel/apic/nmi.c
+++ linux-2.6/arch/x86/kernel/apic/nmi.c
@@ -37,7 +37,6 @@
 
 #include <asm/mach_traps.h>
 
-int unknown_nmi_panic;
 int nmi_watchdog_enabled;
 
 /* For reliability, we're prepared to waste bits here. */
@@ -483,23 +482,6 @@ static void disable_ioapic_nmi_watchdog(
 	on_each_cpu(stop_apic_nmi_watchdog, NULL, 1);
 }
 
-static int __init setup_unknown_nmi_panic(char *str)
-{
-	unknown_nmi_panic = 1;
-	return 1;
-}
-__setup("unknown_nmi_panic", setup_unknown_nmi_panic);
-
-static int unknown_nmi_panic_callback(struct pt_regs *regs, int cpu)
-{
-	unsigned char reason = get_nmi_reason();
-	char buf[64];
-
-	sprintf(buf, "NMI received for unknown reason %02x\n", reason);
-	die_nmi(buf, regs, 1); /* Always panic here */
-	return 0;
-}
-
 /*
  * proc handler for /proc/sys/kernel/nmi
  */
@@ -540,15 +522,6 @@ int proc_nmi_enabled(struct ctl_table *t
 
 #endif /* CONFIG_SYSCTL */
 
-int do_nmi_callback(struct pt_regs *regs, int cpu)
-{
-#ifdef CONFIG_SYSCTL
-	if (unknown_nmi_panic)
-		return unknown_nmi_panic_callback(regs, cpu);
-#endif
-	return 0;
-}
-
 void arch_trigger_all_cpu_backtrace(void)
 {
 	int i;
Index: linux-2.6/arch/x86/kernel/apic/hw_nmi.c
===================================================================
--- linux-2.6.orig/arch/x86/kernel/apic/hw_nmi.c
+++ linux-2.6/arch/x86/kernel/apic/hw_nmi.c
@@ -100,7 +100,6 @@ void acpi_nmi_disable(void) { return; }
 #endif
 atomic_t nmi_active = ATOMIC_INIT(0);           /* oprofile uses this */
 EXPORT_SYMBOL(nmi_active);
-int unknown_nmi_panic;
 void cpu_nmi_set_wd_enabled(void) { return; }
 void stop_apic_nmi_watchdog(void *unused) { return; }
 void setup_apic_nmi_watchdog(void *unused) { return; }
Index: linux-2.6/arch/x86/kernel/traps.c
===================================================================
--- linux-2.6.orig/arch/x86/kernel/traps.c
+++ linux-2.6/arch/x86/kernel/traps.c
@@ -377,6 +377,33 @@ unknown_nmi_error(unsigned char reason,
 	printk(KERN_EMERG "Dazed and confused, but trying to continue\n");
 }
 
+#if defined(CONFIG_SYSCTL) && defined(CONFIG_X86_LOCAL_APIC)
+int unknown_nmi_panic;
+static int __init setup_unknown_nmi_panic(char *str)
+{
+	unknown_nmi_panic = 1;
+	return 1;
+}
+__setup("unknown_nmi_panic", setup_unknown_nmi_panic);
+
+static int unknown_nmi_panic_callback(struct pt_regs *regs, int cpu)
+{
+	unsigned char reason = get_nmi_reason();
+	char buf[64];
+
+	sprintf(buf, "NMI received for unknown reason %02x\n", reason);
+	die_nmi(buf, regs, 1); /* Always panic here */
+	return 0;
+}
+
+static int do_nmi_callback(struct pt_regs *regs, int cpu)
+{
+	if (unknown_nmi_panic)
+		return unknown_nmi_panic_callback(regs, cpu);
+	return 0;
+}
+#endif
+
 static notrace __kprobes void default_do_nmi(struct pt_regs *regs)
 {
 	unsigned char reason = 0;
@@ -405,8 +432,8 @@ static notrace __kprobes void default_do
 		 */
 		if (nmi_watchdog_tick(regs, reason))
 			return;
-		if (!do_nmi_callback(regs, cpu))
 #endif /* !CONFIG_LOCKUP_DETECTOR */
+		if (!do_nmi_callback(regs, cpu))
 			unknown_nmi_error(reason, regs);
 #else
 		unknown_nmi_error(reason, regs);
Index: linux-2.6/kernel/sysctl.c
===================================================================
--- linux-2.6.orig/kernel/sysctl.c
+++ linux-2.6/kernel/sysctl.c
@@ -739,7 +739,7 @@ static struct ctl_table kern_table[] = {
 		.extra2		= &one,
 	},
 #endif
-#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86) && !defined(CONFIG_LOCKUP_DETECTOR)
+#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
 	{
 		.procname       = "unknown_nmi_panic",
 		.data           = &unknown_nmi_panic,
@@ -747,6 +747,8 @@ static struct ctl_table kern_table[] = {
 		.mode           = 0644,
 		.proc_handler   = proc_dointvec,
 	},
+#endif
+#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86) && !defined(CONFIG_LOCKUP_DETECTOR)
 	{
 		.procname       = "nmi_watchdog",
 		.data           = &nmi_watchdog_enabled,
Index: linux-2.6/arch/x86/include/asm/nmi.h
===================================================================
--- linux-2.6.orig/arch/x86/include/asm/nmi.h
+++ linux-2.6/arch/x86/include/asm/nmi.h
@@ -7,14 +7,6 @@
 
 #ifdef ARCH_HAS_NMI_WATCHDOG
 
-/**
- * do_nmi_callback
- *
- * Check to see if a callback exists and execute it.  Return 1
- * if the handler exists and was handled successfully.
- */
-int do_nmi_callback(struct pt_regs *regs, int cpu);
-
 extern void die_nmi(char *str, struct pt_regs *regs, int do_panic);
 extern int check_nmi_watchdog(void);
 #if !defined(CONFIG_LOCKUP_DETECTOR)

Re: [tip:perf/urgent] perf, x86: Fix accidentally ack'ing a second event on intel perf counter

[Don Zickus]

On Fri, Sep 03, 2010 at 12:39:25AM -0700, Yinghai Lu wrote:
> Can you put this into perf branch ?
> 
> Thanks
> 
> Yinghai
> 
> [PATCH] x86,nmi: move unknown_nmi_panic to traps.c

This patch duplicates a bunch of stuff we already have in
unknown_nmi_error.  The only thing I think you are interested in is using
the 'unknown_nmi_panic' flag.  I am putting together a smaller patch that
uses that flag in traps.c (though it would be nice to combine that flag
with panic_on_unrecovered_nmi).

Cheers,
Don

> 
> So we use it even LOCKUP_DETECTOR is defined.
> need Robert/Peter/Don's patch...
> 
> that will keep unknown_nmi_panic to have same behavoir before hw nmi watchdog
> 
> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
> 
> ---
>  arch/x86/include/asm/nmi.h    |    8 --------
>  arch/x86/kernel/apic/hw_nmi.c |    1 -
>  arch/x86/kernel/apic/nmi.c    |   27 ---------------------------
>  arch/x86/kernel/traps.c       |   29 ++++++++++++++++++++++++++++-
>  kernel/sysctl.c               |    4 +++-
>  5 files changed, 31 insertions(+), 38 deletions(-)
> 
> Index: linux-2.6/arch/x86/kernel/apic/nmi.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/kernel/apic/nmi.c
> +++ linux-2.6/arch/x86/kernel/apic/nmi.c
> @@ -37,7 +37,6 @@
>  
>  #include <asm/mach_traps.h>
>  
> -int unknown_nmi_panic;
>  int nmi_watchdog_enabled;
>  
>  /* For reliability, we're prepared to waste bits here. */
> @@ -483,23 +482,6 @@ static void disable_ioapic_nmi_watchdog(
>  	on_each_cpu(stop_apic_nmi_watchdog, NULL, 1);
>  }
>  
> -static int __init setup_unknown_nmi_panic(char *str)
> -{
> -	unknown_nmi_panic = 1;
> -	return 1;
> -}
> -__setup("unknown_nmi_panic", setup_unknown_nmi_panic);
> -
> -static int unknown_nmi_panic_callback(struct pt_regs *regs, int cpu)
> -{
> -	unsigned char reason = get_nmi_reason();
> -	char buf[64];
> -
> -	sprintf(buf, "NMI received for unknown reason %02x\n", reason);
> -	die_nmi(buf, regs, 1); /* Always panic here */
> -	return 0;
> -}
> -
>  /*
>   * proc handler for /proc/sys/kernel/nmi
>   */
> @@ -540,15 +522,6 @@ int proc_nmi_enabled(struct ctl_table *t
>  
>  #endif /* CONFIG_SYSCTL */
>  
> -int do_nmi_callback(struct pt_regs *regs, int cpu)
> -{
> -#ifdef CONFIG_SYSCTL
> -	if (unknown_nmi_panic)
> -		return unknown_nmi_panic_callback(regs, cpu);
> -#endif
> -	return 0;
> -}
> -
>  void arch_trigger_all_cpu_backtrace(void)
>  {
>  	int i;
> Index: linux-2.6/arch/x86/kernel/apic/hw_nmi.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/kernel/apic/hw_nmi.c
> +++ linux-2.6/arch/x86/kernel/apic/hw_nmi.c
> @@ -100,7 +100,6 @@ void acpi_nmi_disable(void) { return; }
>  #endif
>  atomic_t nmi_active = ATOMIC_INIT(0);           /* oprofile uses this */
>  EXPORT_SYMBOL(nmi_active);
> -int unknown_nmi_panic;
>  void cpu_nmi_set_wd_enabled(void) { return; }
>  void stop_apic_nmi_watchdog(void *unused) { return; }
>  void setup_apic_nmi_watchdog(void *unused) { return; }
> Index: linux-2.6/arch/x86/kernel/traps.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/kernel/traps.c
> +++ linux-2.6/arch/x86/kernel/traps.c
> @@ -377,6 +377,33 @@ unknown_nmi_error(unsigned char reason,
>  	printk(KERN_EMERG "Dazed and confused, but trying to continue\n");
>  }
>  
> +#if defined(CONFIG_SYSCTL) && defined(CONFIG_X86_LOCAL_APIC)
> +int unknown_nmi_panic;
> +static int __init setup_unknown_nmi_panic(char *str)
> +{
> +	unknown_nmi_panic = 1;
> +	return 1;
> +}
> +__setup("unknown_nmi_panic", setup_unknown_nmi_panic);
> +
> +static int unknown_nmi_panic_callback(struct pt_regs *regs, int cpu)
> +{
> +	unsigned char reason = get_nmi_reason();
> +	char buf[64];
> +
> +	sprintf(buf, "NMI received for unknown reason %02x\n", reason);
> +	die_nmi(buf, regs, 1); /* Always panic here */
> +	return 0;
> +}
> +
> +static int do_nmi_callback(struct pt_regs *regs, int cpu)
> +{
> +	if (unknown_nmi_panic)
> +		return unknown_nmi_panic_callback(regs, cpu);
> +	return 0;
> +}
> +#endif
> +
>  static notrace __kprobes void default_do_nmi(struct pt_regs *regs)
>  {
>  	unsigned char reason = 0;
> @@ -405,8 +432,8 @@ static notrace __kprobes void default_do
>  		 */
>  		if (nmi_watchdog_tick(regs, reason))
>  			return;
> -		if (!do_nmi_callback(regs, cpu))
>  #endif /* !CONFIG_LOCKUP_DETECTOR */
> +		if (!do_nmi_callback(regs, cpu))
>  			unknown_nmi_error(reason, regs);
>  #else
>  		unknown_nmi_error(reason, regs);
> Index: linux-2.6/kernel/sysctl.c
> ===================================================================
> --- linux-2.6.orig/kernel/sysctl.c
> +++ linux-2.6/kernel/sysctl.c
> @@ -739,7 +739,7 @@ static struct ctl_table kern_table[] = {
>  		.extra2		= &one,
>  	},
>  #endif
> -#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86) && !defined(CONFIG_LOCKUP_DETECTOR)
> +#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
>  	{
>  		.procname       = "unknown_nmi_panic",
>  		.data           = &unknown_nmi_panic,
> @@ -747,6 +747,8 @@ static struct ctl_table kern_table[] = {
>  		.mode           = 0644,
>  		.proc_handler   = proc_dointvec,
>  	},
> +#endif
> +#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86) && !defined(CONFIG_LOCKUP_DETECTOR)
>  	{
>  		.procname       = "nmi_watchdog",
>  		.data           = &nmi_watchdog_enabled,
> Index: linux-2.6/arch/x86/include/asm/nmi.h
> ===================================================================
> --- linux-2.6.orig/arch/x86/include/asm/nmi.h
> +++ linux-2.6/arch/x86/include/asm/nmi.h
> @@ -7,14 +7,6 @@
>  
>  #ifdef ARCH_HAS_NMI_WATCHDOG
>  
> -/**
> - * do_nmi_callback
> - *
> - * Check to see if a callback exists and execute it.  Return 1
> - * if the handler exists and was handled successfully.
> - */
> -int do_nmi_callback(struct pt_regs *regs, int cpu);
> -
>  extern void die_nmi(char *str, struct pt_regs *regs, int do_panic);
>  extern int check_nmi_watchdog(void);
>  #if !defined(CONFIG_LOCKUP_DETECTOR)

Re: [tip:perf/urgent] perf, x86: Fix accidentally ack'ing a second event on intel perf counter

[Yinghai Lu]

On 09/03/2010 08:00 AM, Don Zickus wrote:
> On Fri, Sep 03, 2010 at 12:39:25AM -0700, Yinghai Lu wrote:
>> Can you put this into perf branch ?
>>
>> Thanks
>>
>> Yinghai
>>
>> [PATCH] x86,nmi: move unknown_nmi_panic to traps.c
> 
> This patch duplicates a bunch of stuff we already have in
> unknown_nmi_error.  The only thing I think you are interested in is using
> the 'unknown_nmi_panic' flag.  I am putting together a smaller patch that
> uses that flag in traps.c (though it would be nice to combine that flag
> with panic_on_unrecovered_nmi).

please make sure
keep using unknown_nmi_panic in boot command line and  sysctl
when LOCKUP_DETECTOR is defined.

that does work until hw nmi watchdog is merged with software lock detector.
assume that time hw nmi watchdog is relying on perf nmi and perf nmi would eat all unknown nmi.
good to have Robert/Peter/Don's patches to make per nmi not to eat all unknown nmi.


Thanks

Yinghai

> 
> Cheers,
> Don
> 
>>
>> So we use it even LOCKUP_DETECTOR is defined.
>> need Robert/Peter/Don's patch...
>>
>> that will keep unknown_nmi_panic to have same behavoir before hw nmi watchdog
>>
>> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
>>
>> ---
>>  arch/x86/include/asm/nmi.h    |    8 --------
>>  arch/x86/kernel/apic/hw_nmi.c |    1 -
>>  arch/x86/kernel/apic/nmi.c    |   27 ---------------------------
>>  arch/x86/kernel/traps.c       |   29 ++++++++++++++++++++++++++++-
>>  kernel/sysctl.c               |    4 +++-
>>  5 files changed, 31 insertions(+), 38 deletions(-)
>>
>> Index: linux-2.6/arch/x86/kernel/apic/nmi.c
>> ===================================================================
>> --- linux-2.6.orig/arch/x86/kernel/apic/nmi.c
>> +++ linux-2.6/arch/x86/kernel/apic/nmi.c
>> @@ -37,7 +37,6 @@
>>  
>>  #include <asm/mach_traps.h>
>>  
>> -int unknown_nmi_panic;
>>  int nmi_watchdog_enabled;
>>  
>>  /* For reliability, we're prepared to waste bits here. */
>> @@ -483,23 +482,6 @@ static void disable_ioapic_nmi_watchdog(
>>  	on_each_cpu(stop_apic_nmi_watchdog, NULL, 1);
>>  }
>>  
>> -static int __init setup_unknown_nmi_panic(char *str)
>> -{
>> -	unknown_nmi_panic = 1;
>> -	return 1;
>> -}
>> -__setup("unknown_nmi_panic", setup_unknown_nmi_panic);
>> -
>> -static int unknown_nmi_panic_callback(struct pt_regs *regs, int cpu)
>> -{
>> -	unsigned char reason = get_nmi_reason();
>> -	char buf[64];
>> -
>> -	sprintf(buf, "NMI received for unknown reason %02x\n", reason);
>> -	die_nmi(buf, regs, 1); /* Always panic here */
>> -	return 0;
>> -}
>> -
>>  /*
>>   * proc handler for /proc/sys/kernel/nmi
>>   */
>> @@ -540,15 +522,6 @@ int proc_nmi_enabled(struct ctl_table *t
>>  
>>  #endif /* CONFIG_SYSCTL */
>>  
>> -int do_nmi_callback(struct pt_regs *regs, int cpu)
>> -{
>> -#ifdef CONFIG_SYSCTL
>> -	if (unknown_nmi_panic)
>> -		return unknown_nmi_panic_callback(regs, cpu);
>> -#endif
>> -	return 0;
>> -}
>> -
>>  void arch_trigger_all_cpu_backtrace(void)
>>  {
>>  	int i;
>> Index: linux-2.6/arch/x86/kernel/apic/hw_nmi.c
>> ===================================================================
>> --- linux-2.6.orig/arch/x86/kernel/apic/hw_nmi.c
>> +++ linux-2.6/arch/x86/kernel/apic/hw_nmi.c
>> @@ -100,7 +100,6 @@ void acpi_nmi_disable(void) { return; }
>>  #endif
>>  atomic_t nmi_active = ATOMIC_INIT(0);           /* oprofile uses this */
>>  EXPORT_SYMBOL(nmi_active);
>> -int unknown_nmi_panic;
>>  void cpu_nmi_set_wd_enabled(void) { return; }
>>  void stop_apic_nmi_watchdog(void *unused) { return; }
>>  void setup_apic_nmi_watchdog(void *unused) { return; }
>> Index: linux-2.6/arch/x86/kernel/traps.c
>> ===================================================================
>> --- linux-2.6.orig/arch/x86/kernel/traps.c
>> +++ linux-2.6/arch/x86/kernel/traps.c
>> @@ -377,6 +377,33 @@ unknown_nmi_error(unsigned char reason,
>>  	printk(KERN_EMERG "Dazed and confused, but trying to continue\n");
>>  }
>>  
>> +#if defined(CONFIG_SYSCTL) && defined(CONFIG_X86_LOCAL_APIC)
>> +int unknown_nmi_panic;
>> +static int __init setup_unknown_nmi_panic(char *str)
>> +{
>> +	unknown_nmi_panic = 1;
>> +	return 1;
>> +}
>> +__setup("unknown_nmi_panic", setup_unknown_nmi_panic);
>> +
>> +static int unknown_nmi_panic_callback(struct pt_regs *regs, int cpu)
>> +{
>> +	unsigned char reason = get_nmi_reason();
>> +	char buf[64];
>> +
>> +	sprintf(buf, "NMI received for unknown reason %02x\n", reason);
>> +	die_nmi(buf, regs, 1); /* Always panic here */
>> +	return 0;
>> +}
>> +
>> +static int do_nmi_callback(struct pt_regs *regs, int cpu)
>> +{
>> +	if (unknown_nmi_panic)
>> +		return unknown_nmi_panic_callback(regs, cpu);
>> +	return 0;
>> +}
>> +#endif
>> +
>>  static notrace __kprobes void default_do_nmi(struct pt_regs *regs)
>>  {
>>  	unsigned char reason = 0;
>> @@ -405,8 +432,8 @@ static notrace __kprobes void default_do
>>  		 */
>>  		if (nmi_watchdog_tick(regs, reason))
>>  			return;
>> -		if (!do_nmi_callback(regs, cpu))
>>  #endif /* !CONFIG_LOCKUP_DETECTOR */
>> +		if (!do_nmi_callback(regs, cpu))
>>  			unknown_nmi_error(reason, regs);
>>  #else
>>  		unknown_nmi_error(reason, regs);
>> Index: linux-2.6/kernel/sysctl.c
>> ===================================================================
>> --- linux-2.6.orig/kernel/sysctl.c
>> +++ linux-2.6/kernel/sysctl.c
>> @@ -739,7 +739,7 @@ static struct ctl_table kern_table[] = {
>>  		.extra2		= &one,
>>  	},
>>  #endif
>> -#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86) && !defined(CONFIG_LOCKUP_DETECTOR)
>> +#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
>>  	{
>>  		.procname       = "unknown_nmi_panic",
>>  		.data           = &unknown_nmi_panic,
>> @@ -747,6 +747,8 @@ static struct ctl_table kern_table[] = {
>>  		.mode           = 0644,
>>  		.proc_handler   = proc_dointvec,
>>  	},
>> +#endif
>> +#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86) && !defined(CONFIG_LOCKUP_DETECTOR)
>>  	{
>>  		.procname       = "nmi_watchdog",
>>  		.data           = &nmi_watchdog_enabled,
>> Index: linux-2.6/arch/x86/include/asm/nmi.h
>> ===================================================================
>> --- linux-2.6.orig/arch/x86/include/asm/nmi.h
>> +++ linux-2.6/arch/x86/include/asm/nmi.h
>> @@ -7,14 +7,6 @@
>>  
>>  #ifdef ARCH_HAS_NMI_WATCHDOG
>>  
>> -/**
>> - * do_nmi_callback
>> - *
>> - * Check to see if a callback exists and execute it.  Return 1
>> - * if the handler exists and was handled successfully.
>> - */
>> -int do_nmi_callback(struct pt_regs *regs, int cpu);
>> -
>>  extern void die_nmi(char *str, struct pt_regs *regs, int do_panic);
>>  extern int check_nmi_watchdog(void);
>>  #if !defined(CONFIG_LOCKUP_DETECTOR)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-tip-commits" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [tip:perf/urgent] perf, x86: Fix accidentally ack'ing a second event on intel perf counter

[Don Zickus]

On Fri, Sep 03, 2010 at 10:15:16AM -0700, Yinghai Lu wrote:
> On 09/03/2010 08:00 AM, Don Zickus wrote:
> > On Fri, Sep 03, 2010 at 12:39:25AM -0700, Yinghai Lu wrote:
> >> Can you put this into perf branch ?
> >>
> >> Thanks
> >>
> >> Yinghai
> >>
> >> [PATCH] x86,nmi: move unknown_nmi_panic to traps.c
> > 
> > This patch duplicates a bunch of stuff we already have in
> > unknown_nmi_error.  The only thing I think you are interested in is using
> > the 'unknown_nmi_panic' flag.  I am putting together a smaller patch that
> > uses that flag in traps.c (though it would be nice to combine that flag
> > with panic_on_unrecovered_nmi).
> 
> please make sure
> keep using unknown_nmi_panic in boot command line and  sysctl
> when LOCKUP_DETECTOR is defined.
> 
> that does work until hw nmi watchdog is merged with software lock detector.
> assume that time hw nmi watchdog is relying on perf nmi and perf nmi would eat all unknown nmi.
> good to have Robert/Peter/Don's patches to make per nmi not to eat all unknown nmi.

Hi Yinghai,

Here is the simpler patch I came up with.  It piggy backs off the
unknown_nmi_error code already available.  I compile it with the old and
new nmi watchdog and tested it with sysctl and the kernel parameter.
Everything seems to panic properly.

Let me know if this meets your needs.

Cheers,
Don

diff --git a/arch/x86/kernel/apic/hw_nmi.c b/arch/x86/kernel/apic/hw_nmi.c
index cefd694..e66b16d 100644
--- a/arch/x86/kernel/apic/hw_nmi.c
+++ b/arch/x86/kernel/apic/hw_nmi.c
@@ -100,7 +100,6 @@ void acpi_nmi_disable(void) { return; }
 #endif
 atomic_t nmi_active = ATOMIC_INIT(0);           /* oprofile uses this */
 EXPORT_SYMBOL(nmi_active);
-int unknown_nmi_panic;
 void cpu_nmi_set_wd_enabled(void) { return; }
 void stop_apic_nmi_watchdog(void *unused) { return; }
 void setup_apic_nmi_watchdog(void *unused) { return; }
diff --git a/arch/x86/kernel/apic/nmi.c b/arch/x86/kernel/apic/nmi.c
index a43f71c..dc35af4 100644
--- a/arch/x86/kernel/apic/nmi.c
+++ b/arch/x86/kernel/apic/nmi.c
@@ -37,7 +37,6 @@
 
 #include <asm/mach_traps.h>
 
-int unknown_nmi_panic;
 int nmi_watchdog_enabled;
 
 /* For reliability, we're prepared to waste bits here. */
@@ -483,13 +482,6 @@ static void disable_ioapic_nmi_watchdog(void)
 	on_each_cpu(stop_apic_nmi_watchdog, NULL, 1);
 }
 
-static int __init setup_unknown_nmi_panic(char *str)
-{
-	unknown_nmi_panic = 1;
-	return 1;
-}
-__setup("unknown_nmi_panic", setup_unknown_nmi_panic);
-
 static int unknown_nmi_panic_callback(struct pt_regs *regs, int cpu)
 {
 	unsigned char reason = get_nmi_reason();
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 60788de..095eea8 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -300,6 +300,16 @@ gp_in_kernel:
 	die("general protection fault", regs, error_code);
 }
 
+#if defined(CONFIG_SYSCTL) && defined(CONFIG_X86_LOCAL_APIC)
+int unknown_nmi_panic;
+static int __init setup_unknown_nmi_panic(char *str)
+{
+	unknown_nmi_panic = 1;
+	return 1;
+}
+__setup("unknown_nmi_panic", setup_unknown_nmi_panic);
+#endif
+
 static notrace __kprobes void
 mem_parity_error(unsigned char reason, struct pt_regs *regs)
 {
@@ -371,6 +381,10 @@ unknown_nmi_error(unsigned char reason, struct pt_regs *regs)
 			reason, smp_processor_id());
 
 	printk(KERN_EMERG "Do you have a strange power saving mode enabled?\n");
+#if defined(CONFIG_SYSCTL) && defined(CONFIG_X86_LOCAL_APIC)
+	if (unknown_nmi_panic)
+		die_nmi("", regs, 1);
+#endif
 	if (panic_on_unrecovered_nmi)
 		panic("NMI: Not continuing");
 
diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index ca38e8e..71516a4 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -739,7 +739,7 @@ static struct ctl_table kern_table[] = {
 		.extra2		= &one,
 	},
 #endif
-#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86) && !defined(CONFIG_LOCKUP_DETECTOR)
+#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
 	{
 		.procname       = "unknown_nmi_panic",
 		.data           = &unknown_nmi_panic,
@@ -747,6 +747,8 @@ static struct ctl_table kern_table[] = {
 		.mode           = 0644,
 		.proc_handler   = proc_dointvec,
 	},
+#endif
+#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86) && !defined(CONFIG_LOCKUP_DETECTOR)
 	{
 		.procname       = "nmi_watchdog",
 		.data           = &nmi_watchdog_enabled,

Re: [tip:perf/urgent] perf, x86: Fix accidentally ack'ing a second event on intel perf counter

[Yinghai Lu]

On 09/03/2010 11:35 AM, Don Zickus wrote:
> On Fri, Sep 03, 2010 at 10:15:16AM -0700, Yinghai Lu wrote:
>> On 09/03/2010 08:00 AM, Don Zickus wrote:
>>> On Fri, Sep 03, 2010 at 12:39:25AM -0700, Yinghai Lu wrote:
>>>> Can you put this into perf branch ?
>>>>
>>>> Thanks
>>>>
>>>> Yinghai
>>>>
>>>> [PATCH] x86,nmi: move unknown_nmi_panic to traps.c
>>>
>>> This patch duplicates a bunch of stuff we already have in
>>> unknown_nmi_error.  The only thing I think you are interested in is using
>>> the 'unknown_nmi_panic' flag.  I am putting together a smaller patch that
>>> uses that flag in traps.c (though it would be nice to combine that flag
>>> with panic_on_unrecovered_nmi).
>>
>> please make sure
>> keep using unknown_nmi_panic in boot command line and  sysctl
>> when LOCKUP_DETECTOR is defined.
>>
>> that does work until hw nmi watchdog is merged with software lock detector.
>> assume that time hw nmi watchdog is relying on perf nmi and perf nmi would eat all unknown nmi.
>> good to have Robert/Peter/Don's patches to make per nmi not to eat all unknown nmi.
> 
> Hi Yinghai,
> 
> Here is the simpler patch I came up with.  It piggy backs off the
> unknown_nmi_error code already available.  I compile it with the old and
> new nmi watchdog and tested it with sysctl and the kernel parameter.
> Everything seems to panic properly.
> 
> Let me know if this meets your needs.
it should work. will test later
> 
> Cheers,
> Don
> 
> diff --git a/arch/x86/kernel/apic/hw_nmi.c b/arch/x86/kernel/apic/hw_nmi.c
> index cefd694..e66b16d 100644
> --- a/arch/x86/kernel/apic/hw_nmi.c
> +++ b/arch/x86/kernel/apic/hw_nmi.c
> @@ -100,7 +100,6 @@ void acpi_nmi_disable(void) { return; }
>  #endif
>  atomic_t nmi_active = ATOMIC_INIT(0);           /* oprofile uses this */
>  EXPORT_SYMBOL(nmi_active);
> -int unknown_nmi_panic;
>  void cpu_nmi_set_wd_enabled(void) { return; }
>  void stop_apic_nmi_watchdog(void *unused) { return; }
>  void setup_apic_nmi_watchdog(void *unused) { return; }
> diff --git a/arch/x86/kernel/apic/nmi.c b/arch/x86/kernel/apic/nmi.c
> index a43f71c..dc35af4 100644
> --- a/arch/x86/kernel/apic/nmi.c
> +++ b/arch/x86/kernel/apic/nmi.c
> @@ -37,7 +37,6 @@
>  
>  #include <asm/mach_traps.h>
>  
> -int unknown_nmi_panic;
>  int nmi_watchdog_enabled;
>  
>  /* For reliability, we're prepared to waste bits here. */
> @@ -483,13 +482,6 @@ static void disable_ioapic_nmi_watchdog(void)
>  	on_each_cpu(stop_apic_nmi_watchdog, NULL, 1);
>  }
>  
> -static int __init setup_unknown_nmi_panic(char *str)
> -{
> -	unknown_nmi_panic = 1;
> -	return 1;
> -}
> -__setup("unknown_nmi_panic", setup_unknown_nmi_panic);
> -
>  static int unknown_nmi_panic_callback(struct pt_regs *regs, int cpu)
>  {
>  	unsigned char reason = get_nmi_reason();
> diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
> index 60788de..095eea8 100644
> --- a/arch/x86/kernel/traps.c
> +++ b/arch/x86/kernel/traps.c
> @@ -300,6 +300,16 @@ gp_in_kernel:
>  	die("general protection fault", regs, error_code);
>  }
>  
> +#if defined(CONFIG_SYSCTL) && defined(CONFIG_X86_LOCAL_APIC)
> +int unknown_nmi_panic;
> +static int __init setup_unknown_nmi_panic(char *str)
> +{
> +	unknown_nmi_panic = 1;
> +	return 1;
> +}
> +__setup("unknown_nmi_panic", setup_unknown_nmi_panic);
> +#endif
> +
>  static notrace __kprobes void
>  mem_parity_error(unsigned char reason, struct pt_regs *regs)
>  {
> @@ -371,6 +381,10 @@ unknown_nmi_error(unsigned char reason, struct pt_regs *regs)
>  			reason, smp_processor_id());
>  
>  	printk(KERN_EMERG "Do you have a strange power saving mode enabled?\n");
> +#if defined(CONFIG_SYSCTL) && defined(CONFIG_X86_LOCAL_APIC)
> +	if (unknown_nmi_panic)
> +		die_nmi("", regs, 1);
> +#endif

can you move it before the two printk(KERN_EMERG...)?

>  	if (panic_on_unrecovered_nmi)
>  		panic("NMI: Not continuing");

Can you merge those two panic? 

Thanks

Yinghai Lu

>  
> diff --git a/kernel/sysctl.c b/kernel/sysctl.c
> index ca38e8e..71516a4 100644
> --- a/kernel/sysctl.c
> +++ b/kernel/sysctl.c
> @@ -739,7 +739,7 @@ static struct ctl_table kern_table[] = {
>  		.extra2		= &one,
>  	},
>  #endif
> -#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86) && !defined(CONFIG_LOCKUP_DETECTOR)
> +#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
>  	{
>  		.procname       = "unknown_nmi_panic",
>  		.data           = &unknown_nmi_panic,
> @@ -747,6 +747,8 @@ static struct ctl_table kern_table[] = {
>  		.mode           = 0644,
>  		.proc_handler   = proc_dointvec,
>  	},
> +#endif
> +#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86) && !defined(CONFIG_LOCKUP_DETECTOR)
>  	{
>  		.procname       = "nmi_watchdog",
>  		.data           = &nmi_watchdog_enabled,
> --
> To unsubscribe from this list: send the line "unsubscribe linux-tip-commits" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [tip:perf/urgent] perf, x86: Fix accidentally ack'ing a second event on intel perf counter

[Don Zickus]

On Fri, Sep 03, 2010 at 12:24:09PM -0700, Yinghai Lu wrote:
> > Let me know if this meets your needs.
> it should work. will test later
> > 
> > Cheers,
> > Don
> > 
> > @@ -371,6 +381,10 @@ unknown_nmi_error(unsigned char reason, struct pt_regs *regs)
> >  			reason, smp_processor_id());
> >  
> >  	printk(KERN_EMERG "Do you have a strange power saving mode enabled?\n");
> > +#if defined(CONFIG_SYSCTL) && defined(CONFIG_X86_LOCAL_APIC)
> > +	if (unknown_nmi_panic)
> > +		die_nmi("", regs, 1);
> > +#endif
> 
> can you move it before the two printk(KERN_EMERG...)?

Well, then you do not print a reason for the panic.  Though the one printk
might not be useful.

> 
> >  	if (panic_on_unrecovered_nmi)
> >  		panic("NMI: Not continuing");
> 
> Can you merge those two panic? 

I thought about it but did not want to disturb the panic vs die_nmi paths.
But I can definitely do that I think.

Cheers,
Don

Re: [tip:perf/urgent] perf, x86: Fix accidentally ack'ing a second event on intel perf counter

[Yinghai Lu]

On 09/03/2010 11:35 AM, Don Zickus wrote:
> On Fri, Sep 03, 2010 at 10:15:16AM -0700, Yinghai Lu wrote:
>> On 09/03/2010 08:00 AM, Don Zickus wrote:
>>>>
>>>> [PATCH] x86,nmi: move unknown_nmi_panic to traps.c
>>>
>>> This patch duplicates a bunch of stuff we already have in
>>> unknown_nmi_error.  The only thing I think you are interested in is using
>>> the 'unknown_nmi_panic' flag.  I am putting together a smaller patch that
>>> uses that flag in traps.c (though it would be nice to combine that flag
>>> with panic_on_unrecovered_nmi).
>>
>> please make sure
>> keep using unknown_nmi_panic in boot command line and  sysctl
>> when LOCKUP_DETECTOR is defined.
>>
>> that does work until hw nmi watchdog is merged with software lock detector.
>> assume that time hw nmi watchdog is relying on perf nmi and perf nmi would eat all unknown nmi.
>> good to have Robert/Peter/Don's patches to make per nmi not to eat all unknown nmi.
> 
> Hi Yinghai,
> 
> Here is the simpler patch I came up with.  It piggy backs off the
> unknown_nmi_error code already available.  I compile it with the old and
> new nmi watchdog and tested it with sysctl and the kernel parameter.
> Everything seems to panic properly.
> 
> Let me know if this meets your needs.


...

> 
> diff --git a/arch/x86/kernel/apic/hw_nmi.c b/arch/x86/kernel/apic/hw_nmi.c
> index cefd694..e66b16d 100644
> --- a/arch/x86/kernel/apic/hw_nmi.c
> +++ b/arch/x86/kernel/apic/hw_nmi.c
> @@ -100,7 +100,6 @@ void acpi_nmi_disable(void) { return; }
>  #endif
>  atomic_t nmi_active = ATOMIC_INIT(0);           /* oprofile uses this */
>  EXPORT_SYMBOL(nmi_active);
> -int unknown_nmi_panic;
>  void cpu_nmi_set_wd_enabled(void) { return; }
>  void stop_apic_nmi_watchdog(void *unused) { return; }
>  void setup_apic_nmi_watchdog(void *unused) { return; }
> diff --git a/arch/x86/kernel/apic/nmi.c b/arch/x86/kernel/apic/nmi.c
> index a43f71c..dc35af4 100644
> --- a/arch/x86/kernel/apic/nmi.c
> +++ b/arch/x86/kernel/apic/nmi.c
> @@ -37,7 +37,6 @@
>  
>  #include <asm/mach_traps.h>
>  
> -int unknown_nmi_panic;
>  int nmi_watchdog_enabled;
>  
>  /* For reliability, we're prepared to waste bits here. */
> @@ -483,13 +482,6 @@ static void disable_ioapic_nmi_watchdog(void)
>  	on_each_cpu(stop_apic_nmi_watchdog, NULL, 1);
>  }
>  
> -static int __init setup_unknown_nmi_panic(char *str)
> -{
> -	unknown_nmi_panic = 1;
> -	return 1;
> -}
> -__setup("unknown_nmi_panic", setup_unknown_nmi_panic);
> -
>  static int unknown_nmi_panic_callback(struct pt_regs *regs, int cpu)
>  {
>  	unsigned char reason = get_nmi_reason();
> diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
> index 60788de..095eea8 100644
> --- a/arch/x86/kernel/traps.c
> +++ b/arch/x86/kernel/traps.c
> @@ -300,6 +300,16 @@ gp_in_kernel:
>  	die("general protection fault", regs, error_code);
>  }
>  
> +#if defined(CONFIG_SYSCTL) && defined(CONFIG_X86_LOCAL_APIC)
> +int unknown_nmi_panic;
> +static int __init setup_unknown_nmi_panic(char *str)
> +{
> +	unknown_nmi_panic = 1;
> +	return 1;
> +}
> +__setup("unknown_nmi_panic", setup_unknown_nmi_panic);
> +#endif
> +
>  static notrace __kprobes void
>  mem_parity_error(unsigned char reason, struct pt_regs *regs)
>  {
> @@ -371,6 +381,10 @@ unknown_nmi_error(unsigned char reason, struct pt_regs *regs)
>  			reason, smp_processor_id());
>  
>  	printk(KERN_EMERG "Do you have a strange power saving mode enabled?\n");
> +#if defined(CONFIG_SYSCTL) && defined(CONFIG_X86_LOCAL_APIC)
> +	if (unknown_nmi_panic)
> +		die_nmi("", regs, 1);
> +#endif
>  	if (panic_on_unrecovered_nmi)
>  		panic("NMI: Not continuing");

please consider to move 
	printk(KERN_EMERG "Do you have a strange power saving mode enabled?\n");
down.

test it, and it works on my test setups.

please submit complete patch to Ingo.

Thanks

Yinghai


>  
> diff --git a/kernel/sysctl.c b/kernel/sysctl.c
> index ca38e8e..71516a4 100644
> --- a/kernel/sysctl.c
> +++ b/kernel/sysctl.c
> @@ -739,7 +739,7 @@ static struct ctl_table kern_table[] = {
>  		.extra2		= &one,
>  	},
>  #endif
> -#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86) && !defined(CONFIG_LOCKUP_DETECTOR)
> +#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
>  	{
>  		.procname       = "unknown_nmi_panic",
>  		.data           = &unknown_nmi_panic,
> @@ -747,6 +747,8 @@ static struct ctl_table kern_table[] = {
>  		.mode           = 0644,
>  		.proc_handler   = proc_dointvec,
>  	},
> +#endif
> +#if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86) && !defined(CONFIG_LOCKUP_DETECTOR)
>  	{
>  		.procname       = "nmi_watchdog",
>  		.data           = &nmi_watchdog_enabled,
> --
> To unsubscribe from this list: send the line "unsubscribe linux-tip-commits" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [tip:perf/urgent] perf, x86: Fix accidentally ack'ing a second event on intel perf counter

[Don Zickus]

On Mon, Oct 04, 2010 at 04:24:14PM -0700, Yinghai Lu wrote:
> >  	printk(KERN_EMERG "Do you have a strange power saving mode enabled?\n");
> > +#if defined(CONFIG_SYSCTL) && defined(CONFIG_X86_LOCAL_APIC)
> > +	if (unknown_nmi_panic)
> > +		die_nmi("", regs, 1);
> > +#endif
> >  	if (panic_on_unrecovered_nmi)
> >  		panic("NMI: Not continuing");
> 
> please consider to move 
> 	printk(KERN_EMERG "Do you have a strange power saving mode enabled?\n");
> down.
> 
> test it, and it works on my test setups.
> 
> please submit complete patch to Ingo.

I think Huang Ying from Intel has a patchset with similar changes.  I'll
have him make the changes.

Cheers,
Don

[PATCH 2/3] perf, x86: Try to handle unknown nmis with an enabled PMU

[Don Zickus]

From: Robert Richter <robert.richter@amd.com>

When the PMU is enabled it is valid to have unhandled nmis, two
events could trigger 'simultaneously' raising two back-to-back
NMIs. If the first NMI handles both, the latter will be empty and daze
the CPU.

The solution to avoid an 'unknown nmi' massage in this case was simply
to stop the nmi handler chain when the PMU is enabled by stating
the nmi was handled. This has the drawback that a) we can not detect
unknown nmis anymore, and b) subsequent nmi handlers are not called.

This patch addresses this. Now, we check this unknown NMI if it could
be a PMU back-to-back NMI. Otherwise we pass it and let the kernel
handle the unknown nmi.

This is a debug log:

cpu #6, nmi #32333, skip_nmi #32330, handled = 1, time = 1934364430
cpu #6, nmi #32334, skip_nmi #32330, handled = 1, time = 1934704616
cpu #6, nmi #32335, skip_nmi #32336, handled = 2, time = 1936032320
cpu #6, nmi #32336, skip_nmi #32336, handled = 0, time = 1936034139
cpu #6, nmi #32337, skip_nmi #32336, handled = 1, time = 1936120100
cpu #6, nmi #32338, skip_nmi #32336, handled = 1, time = 1936404607
cpu #6, nmi #32339, skip_nmi #32336, handled = 1, time = 1937983416
cpu #6, nmi #32340, skip_nmi #32341, handled = 2, time = 1938201032
cpu #6, nmi #32341, skip_nmi #32341, handled = 0, time = 1938202830
cpu #6, nmi #32342, skip_nmi #32341, handled = 1, time = 1938443743
cpu #6, nmi #32343, skip_nmi #32341, handled = 1, time = 1939956552
cpu #6, nmi #32344, skip_nmi #32341, handled = 1, time = 1940073224
cpu #6, nmi #32345, skip_nmi #32341, handled = 1, time = 1940485677
cpu #6, nmi #32346, skip_nmi #32347, handled = 2, time = 1941947772
cpu #6, nmi #32347, skip_nmi #32347, handled = 1, time = 1941949818
cpu #6, nmi #32348, skip_nmi #32347, handled = 0, time = 1941951591
Uhhuh. NMI received for unknown reason 00 on CPU 6.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue

Deltas:

nmi #32334 340186
nmi #32335 1327704
nmi #32336 1819      <<<< back-to-back nmi [1]
nmi #32337 85961
nmi #32338 284507
nmi #32339 1578809
nmi #32340 217616
nmi #32341 1798      <<<< back-to-back nmi [2]
nmi #32342 240913
nmi #32343 1512809
nmi #32344 116672
nmi #32345 412453
nmi #32346 1462095   <<<< 1st nmi (standard) handling 2 counters
nmi #32347 2046      <<<< 2nd nmi (back-to-back) handling one counter
nmi #32348 1773      <<<< 3rd nmi (back-to-back) handling no counter! [3]

For  back-to-back nmi detection there are the following rules:

The PMU nmi handler was handling more than one counter and no
counter was handled in the subsequent nmi (see [1] and [2] above).

There is another case if there are two subsequent back-to-back nmis
[3]. The 2nd is detected as back-to-back because the first handled
more than one counter. If the second handles one counter and the 3rd
handles nothing, we drop the 3rd nmi because it could be a
back-to-back nmi.

[ dz - renamed nmi variable to pmu_nmi to avoid clash with .nmi in
  entry.S ]

Signed-off-by: Robert Richter <robert.richter@amd.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <20100817152225.GQ26154@erda.amd.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Don Zickus <dzickus@redhat.com>
---
 arch/x86/kernel/cpu/perf_event.c |   59 +++++++++++++++++++++++++++++--------
 1 files changed, 46 insertions(+), 13 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index f2da20f..3efdf28 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1154,7 +1154,7 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 		/*
 		 * event overflow
 		 */
-		handled		= 1;
+		handled++;
 		data.period	= event->hw.last_period;
 
 		if (!x86_perf_event_set_period(event))
@@ -1200,12 +1200,20 @@ void perf_events_lapic_init(void)
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 }
 
+struct pmu_nmi_state {
+	unsigned int	marked;
+	int		handled;
+};
+
+static DEFINE_PER_CPU(struct pmu_nmi_state, pmu_nmi);
+
 static int __kprobes
 perf_event_nmi_handler(struct notifier_block *self,
 			 unsigned long cmd, void *__args)
 {
 	struct die_args *args = __args;
-	struct pt_regs *regs;
+	unsigned int this_nmi;
+	int handled;
 
 	if (!atomic_read(&active_events))
 		return NOTIFY_DONE;
@@ -1214,22 +1222,47 @@ perf_event_nmi_handler(struct notifier_block *self,
 	case DIE_NMI:
 	case DIE_NMI_IPI:
 		break;
-
+	case DIE_NMIUNKNOWN:
+		this_nmi = percpu_read(irq_stat.__nmi_count);
+		if (this_nmi != __get_cpu_var(pmu_nmi).marked)
+			/* let the kernel handle the unknown nmi */
+			return NOTIFY_DONE;
+		/*
+		 * This one is a PMU back-to-back nmi. Two events
+		 * trigger 'simultaneously' raising two back-to-back
+		 * NMIs. If the first NMI handles both, the latter
+		 * will be empty and daze the CPU. So, we drop it to
+		 * avoid false-positive 'unknown nmi' messages.
+		 */
+		return NOTIFY_STOP;
 	default:
 		return NOTIFY_DONE;
 	}
 
-	regs = args->regs;
-
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
-	/*
-	 * Can't rely on the handled return value to say it was our NMI, two
-	 * events could trigger 'simultaneously' raising two back-to-back NMIs.
-	 *
-	 * If the first NMI handles both, the latter will be empty and daze
-	 * the CPU.
-	 */
-	x86_pmu.handle_irq(regs);
+
+	handled = x86_pmu.handle_irq(args->regs);
+	if (!handled)
+		return NOTIFY_DONE;
+
+	this_nmi = percpu_read(irq_stat.__nmi_count);
+	if ((handled > 1) ||
+		/* the next nmi could be a back-to-back nmi */
+	    ((__get_cpu_var(pmu_nmi).marked == this_nmi) &&
+	     (__get_cpu_var(pmu_nmi).handled > 1))) {
+		/*
+		 * We could have two subsequent back-to-back nmis: The
+		 * first handles more than one counter, the 2nd
+		 * handles only one counter and the 3rd handles no
+		 * counter.
+		 *
+		 * This is the 2nd nmi because the previous was
+		 * handling more than one counter. We will mark the
+		 * next (3rd) and then drop it if unhandled.
+		 */
+		__get_cpu_var(pmu_nmi).marked	= this_nmi + 1;
+		__get_cpu_var(pmu_nmi).handled	= handled;
+	}
 
 	return NOTIFY_STOP;
 }
-- 
1.7.2.2

[tip:perf/urgent] perf, x86: Try to handle unknown nmis with an enabled PMU

[tip-bot for Robert Richter]

Commit-ID:  4177c42a6301a34c20038ec2771a33dcc30bb338
Gitweb:     http://git.kernel.org/tip/4177c42a6301a34c20038ec2771a33dcc30bb338
Author:     Robert Richter <robert.richter@amd.com>
AuthorDate: Thu, 2 Sep 2010 15:07:48 -0400
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 3 Sep 2010 08:05:18 +0200

perf, x86: Try to handle unknown nmis with an enabled PMU

When the PMU is enabled it is valid to have unhandled nmis, two
events could trigger 'simultaneously' raising two back-to-back
NMIs. If the first NMI handles both, the latter will be empty
and daze the CPU.

The solution to avoid an 'unknown nmi' massage in this case was
simply to stop the nmi handler chain when the PMU is enabled by
stating the nmi was handled. This has the drawback that a) we
can not detect unknown nmis anymore, and b) subsequent nmi
handlers are not called.

This patch addresses this. Now, we check this unknown NMI if it
could be a PMU back-to-back NMI. Otherwise we pass it and let
the kernel handle the unknown nmi.

This is a debug log:

 cpu #6, nmi #32333, skip_nmi #32330, handled = 1, time = 1934364430
 cpu #6, nmi #32334, skip_nmi #32330, handled = 1, time = 1934704616
 cpu #6, nmi #32335, skip_nmi #32336, handled = 2, time = 1936032320
 cpu #6, nmi #32336, skip_nmi #32336, handled = 0, time = 1936034139
 cpu #6, nmi #32337, skip_nmi #32336, handled = 1, time = 1936120100
 cpu #6, nmi #32338, skip_nmi #32336, handled = 1, time = 1936404607
 cpu #6, nmi #32339, skip_nmi #32336, handled = 1, time = 1937983416
 cpu #6, nmi #32340, skip_nmi #32341, handled = 2, time = 1938201032
 cpu #6, nmi #32341, skip_nmi #32341, handled = 0, time = 1938202830
 cpu #6, nmi #32342, skip_nmi #32341, handled = 1, time = 1938443743
 cpu #6, nmi #32343, skip_nmi #32341, handled = 1, time = 1939956552
 cpu #6, nmi #32344, skip_nmi #32341, handled = 1, time = 1940073224
 cpu #6, nmi #32345, skip_nmi #32341, handled = 1, time = 1940485677
 cpu #6, nmi #32346, skip_nmi #32347, handled = 2, time = 1941947772
 cpu #6, nmi #32347, skip_nmi #32347, handled = 1, time = 1941949818
 cpu #6, nmi #32348, skip_nmi #32347, handled = 0, time = 1941951591
 Uhhuh. NMI received for unknown reason 00 on CPU 6.
 Do you have a strange power saving mode enabled?
 Dazed and confused, but trying to continue

Deltas:

 nmi #32334 340186
 nmi #32335 1327704
 nmi #32336 1819      <<<< back-to-back nmi [1]
 nmi #32337 85961
 nmi #32338 284507
 nmi #32339 1578809
 nmi #32340 217616
 nmi #32341 1798      <<<< back-to-back nmi [2]
 nmi #32342 240913
 nmi #32343 1512809
 nmi #32344 116672
 nmi #32345 412453
 nmi #32346 1462095   <<<< 1st nmi (standard) handling 2 counters
 nmi #32347 2046      <<<< 2nd nmi (back-to-back) handling one
 counter nmi #32348 1773      <<<< 3rd nmi (back-to-back)
 handling no counter! [3]

For  back-to-back nmi detection there are the following rules:

The PMU nmi handler was handling more than one counter and no
counter was handled in the subsequent nmi (see [1] and [2]
above).

There is another case if there are two subsequent back-to-back
nmis [3]. The 2nd is detected as back-to-back because the first
handled more than one counter. If the second handles one counter
and the 3rd handles nothing, we drop the 3rd nmi because it
could be a back-to-back nmi.

Signed-off-by: Robert Richter <robert.richter@amd.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
[ renamed nmi variable to pmu_nmi to avoid clash with .nmi in entry.S ]
Signed-off-by: Don Zickus <dzickus@redhat.com>
Cc: peterz@infradead.org
Cc: gorcunov@gmail.com
Cc: fweisbec@gmail.com
Cc: ying.huang@intel.com
Cc: ming.m.lin@intel.com
Cc: eranian@google.com
LKML-Reference: <1283454469-1909-3-git-send-email-dzickus@redhat.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 arch/x86/kernel/cpu/perf_event.c |   59 +++++++++++++++++++++++++++++--------
 1 files changed, 46 insertions(+), 13 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index f2da20f..3efdf28 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1154,7 +1154,7 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 		/*
 		 * event overflow
 		 */
-		handled		= 1;
+		handled++;
 		data.period	= event->hw.last_period;
 
 		if (!x86_perf_event_set_period(event))
@@ -1200,12 +1200,20 @@ void perf_events_lapic_init(void)
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 }
 
+struct pmu_nmi_state {
+	unsigned int	marked;
+	int		handled;
+};
+
+static DEFINE_PER_CPU(struct pmu_nmi_state, pmu_nmi);
+
 static int __kprobes
 perf_event_nmi_handler(struct notifier_block *self,
 			 unsigned long cmd, void *__args)
 {
 	struct die_args *args = __args;
-	struct pt_regs *regs;
+	unsigned int this_nmi;
+	int handled;
 
 	if (!atomic_read(&active_events))
 		return NOTIFY_DONE;
@@ -1214,22 +1222,47 @@ perf_event_nmi_handler(struct notifier_block *self,
 	case DIE_NMI:
 	case DIE_NMI_IPI:
 		break;
-
+	case DIE_NMIUNKNOWN:
+		this_nmi = percpu_read(irq_stat.__nmi_count);
+		if (this_nmi != __get_cpu_var(pmu_nmi).marked)
+			/* let the kernel handle the unknown nmi */
+			return NOTIFY_DONE;
+		/*
+		 * This one is a PMU back-to-back nmi. Two events
+		 * trigger 'simultaneously' raising two back-to-back
+		 * NMIs. If the first NMI handles both, the latter
+		 * will be empty and daze the CPU. So, we drop it to
+		 * avoid false-positive 'unknown nmi' messages.
+		 */
+		return NOTIFY_STOP;
 	default:
 		return NOTIFY_DONE;
 	}
 
-	regs = args->regs;
-
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
-	/*
-	 * Can't rely on the handled return value to say it was our NMI, two
-	 * events could trigger 'simultaneously' raising two back-to-back NMIs.
-	 *
-	 * If the first NMI handles both, the latter will be empty and daze
-	 * the CPU.
-	 */
-	x86_pmu.handle_irq(regs);
+
+	handled = x86_pmu.handle_irq(args->regs);
+	if (!handled)
+		return NOTIFY_DONE;
+
+	this_nmi = percpu_read(irq_stat.__nmi_count);
+	if ((handled > 1) ||
+		/* the next nmi could be a back-to-back nmi */
+	    ((__get_cpu_var(pmu_nmi).marked == this_nmi) &&
+	     (__get_cpu_var(pmu_nmi).handled > 1))) {
+		/*
+		 * We could have two subsequent back-to-back nmis: The
+		 * first handles more than one counter, the 2nd
+		 * handles only one counter and the 3rd handles no
+		 * counter.
+		 *
+		 * This is the 2nd nmi because the previous was
+		 * handling more than one counter. We will mark the
+		 * next (3rd) and then drop it if unhandled.
+		 */
+		__get_cpu_var(pmu_nmi).marked	= this_nmi + 1;
+		__get_cpu_var(pmu_nmi).handled	= handled;
+	}
 
 	return NOTIFY_STOP;
 }

[PATCH 3/3] perf, x86: Fix handle_irq return values

[Don Zickus]

From: Peter Zijlstra <a.p.zijlstra@chello.nl>

Now that we rely on the number of handled overflows, ensure all handle_irq
implementations actually return the right number.

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <1282228033.2605.204.camel@laptop>
Signed-off-by: Don Zickus <dzickus@redhat.com>
---
 arch/x86/kernel/cpu/perf_event_intel.c |    9 +++++++--
 arch/x86/kernel/cpu/perf_event_p4.c    |    2 +-
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
index 1297bf1..ee05c90 100644
--- a/arch/x86/kernel/cpu/perf_event_intel.c
+++ b/arch/x86/kernel/cpu/perf_event_intel.c
@@ -713,6 +713,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	struct cpu_hw_events *cpuc;
 	int bit, loops;
 	u64 status;
+	int handled = 0;
 
 	perf_sample_data_init(&data, 0);
 
@@ -743,12 +744,16 @@ again:
 	/*
 	 * PEBS overflow sets bit 62 in the global status register
 	 */
-	if (__test_and_clear_bit(62, (unsigned long *)&status))
+	if (__test_and_clear_bit(62, (unsigned long *)&status)) {
+		handled++;
 		x86_pmu.drain_pebs(regs);
+	}
 
 	for_each_set_bit(bit, (unsigned long *)&status, X86_PMC_IDX_MAX) {
 		struct perf_event *event = cpuc->events[bit];
 
+		handled++;
+
 		if (!test_bit(bit, cpuc->active_mask))
 			continue;
 
@@ -770,7 +775,7 @@ again:
 
 done:
 	intel_pmu_enable_all(0);
-	return 1;
+	return handled;
 }
 
 static struct event_constraint *
diff --git a/arch/x86/kernel/cpu/perf_event_p4.c b/arch/x86/kernel/cpu/perf_event_p4.c
index febb12c..d470c91 100644
--- a/arch/x86/kernel/cpu/perf_event_p4.c
+++ b/arch/x86/kernel/cpu/perf_event_p4.c
@@ -690,7 +690,7 @@ static int p4_pmu_handle_irq(struct pt_regs *regs)
 		inc_irq_stat(apic_perf_irqs);
 	}
 
-	return handled > 0;
+	return handled;
 }
 
 /*
-- 
1.7.2.2

[tip:perf/urgent] perf, x86: Fix handle_irq return values

[tip-bot for Peter Zijlstra]

Commit-ID:  de725dec9de7a7541996176d59cf8542365b8b0e
Gitweb:     http://git.kernel.org/tip/de725dec9de7a7541996176d59cf8542365b8b0e
Author:     Peter Zijlstra <a.p.zijlstra@chello.nl>
AuthorDate: Thu, 2 Sep 2010 15:07:49 -0400
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 3 Sep 2010 08:05:18 +0200

perf, x86: Fix handle_irq return values

Now that we rely on the number of handled overflows, ensure all
handle_irq implementations actually return the right number.

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Don Zickus <dzickus@redhat.com>
Cc: peterz@infradead.org
Cc: robert.richter@amd.com
Cc: gorcunov@gmail.com
Cc: fweisbec@gmail.com
Cc: ying.huang@intel.com
Cc: ming.m.lin@intel.com
Cc: eranian@google.com
LKML-Reference: <1283454469-1909-4-git-send-email-dzickus@redhat.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 arch/x86/kernel/cpu/perf_event_intel.c |    9 +++++++--
 arch/x86/kernel/cpu/perf_event_p4.c    |    2 +-
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c
index 1297bf1..ee05c90 100644
--- a/arch/x86/kernel/cpu/perf_event_intel.c
+++ b/arch/x86/kernel/cpu/perf_event_intel.c
@@ -713,6 +713,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
 	struct cpu_hw_events *cpuc;
 	int bit, loops;
 	u64 status;
+	int handled = 0;
 
 	perf_sample_data_init(&data, 0);
 
@@ -743,12 +744,16 @@ again:
 	/*
 	 * PEBS overflow sets bit 62 in the global status register
 	 */
-	if (__test_and_clear_bit(62, (unsigned long *)&status))
+	if (__test_and_clear_bit(62, (unsigned long *)&status)) {
+		handled++;
 		x86_pmu.drain_pebs(regs);
+	}
 
 	for_each_set_bit(bit, (unsigned long *)&status, X86_PMC_IDX_MAX) {
 		struct perf_event *event = cpuc->events[bit];
 
+		handled++;
+
 		if (!test_bit(bit, cpuc->active_mask))
 			continue;
 
@@ -770,7 +775,7 @@ again:
 
 done:
 	intel_pmu_enable_all(0);
-	return 1;
+	return handled;
 }
 
 static struct event_constraint *
diff --git a/arch/x86/kernel/cpu/perf_event_p4.c b/arch/x86/kernel/cpu/perf_event_p4.c
index 7e578e9..b560db3 100644
--- a/arch/x86/kernel/cpu/perf_event_p4.c
+++ b/arch/x86/kernel/cpu/perf_event_p4.c
@@ -692,7 +692,7 @@ static int p4_pmu_handle_irq(struct pt_regs *regs)
 		inc_irq_stat(apic_perf_irqs);
 	}
 
-	return handled > 0;
+	return handled;
 }
 
 /*

Re: [PATCH 0/3 v2] nmi perf fixes

[Peter Zijlstra]

On Thu, 2010-09-02 at 15:07 -0400, Don Zickus wrote:
> Fixes to allow unknown nmis to pass through the perf nmi handler instead
> of being swallowed.  Contains patches that are already in Ingo's tree.  Added
> here for completeness.  Based on ingo/tip
> 
> Tested on intel/amd
> 
> v2: patch cleanups and consolidation, no code changes
> 
> Don Zickus (1):
>   perf, x86: Fix accidentally ack'ing a second event on intel perf
>     counter
> 
> Peter Zijlstra (1):
>   perf, x86: Fix handle_irq return values
> 
> Robert Richter (1):
>   perf, x86: Try to handle unknown nmis with an enabled PMU
> 
>  arch/x86/kernel/cpu/perf_event.c       |   59 +++++++++++++++++++++++++-------
>  arch/x86/kernel/cpu/perf_event_intel.c |   15 +++++---
>  arch/x86/kernel/cpu/perf_event_p4.c    |    2 +-
>  3 files changed, 56 insertions(+), 20 deletions(-)

Both Ingo and I are getting Dazed and confused on our AMD machines, it
started before yesterday (that is, after backing out all my recent
changes it still gets dazed), so I suspect this set.

I'll look at getting a trace of the thing, but if any of you has a
bright idea...

Re: [PATCH 0/3 v2] nmi perf fixes

[Stephane Eranian]

On Fri, Sep 10, 2010 at 1:41 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Thu, 2010-09-02 at 15:07 -0400, Don Zickus wrote:
>> Fixes to allow unknown nmis to pass through the perf nmi handler instead
>> of being swallowed.  Contains patches that are already in Ingo's tree.  Added
>> here for completeness.  Based on ingo/tip
>>
>> Tested on intel/amd
>>
>> v2: patch cleanups and consolidation, no code changes
>>
>> Don Zickus (1):
>>   perf, x86: Fix accidentally ack'ing a second event on intel perf
>>     counter
>>
>> Peter Zijlstra (1):
>>   perf, x86: Fix handle_irq return values
>>
>> Robert Richter (1):
>>   perf, x86: Try to handle unknown nmis with an enabled PMU
>>
>>  arch/x86/kernel/cpu/perf_event.c       |   59 +++++++++++++++++++++++++-------
>>  arch/x86/kernel/cpu/perf_event_intel.c |   15 +++++---
>>  arch/x86/kernel/cpu/perf_event_p4.c    |    2 +-
>>  3 files changed, 56 insertions(+), 20 deletions(-)
>
> Both Ingo and I are getting Dazed and confused on our AMD machines, it
> started before yesterday (that is, after backing out all my recent
> changes it still gets dazed), so I suspect this set.
>
> I'll look at getting a trace of the thing, but if any of you has a
> bright idea...

I still don't buy the back-to-back NMI thing. I suspect there is
something else going on. I have continued to track it down.
I got closer yesterday, until I ran into other issues. It may
have to do with throttling. I am still trying to understanding
why the OVF_STATUS does not match the check based on
the counter values.

>

Re: [PATCH 0/3 v2] nmi perf fixes

[Stephane Eranian]

I think the broader issue here is that I think we should not have
so many subsystems hanging off of that die_register(). The NMI
should be for perf_events only. Many in there have nothing to do
with performance monitoring. They are mostly debug features.

On Fri, Sep 10, 2010 at 2:10 PM, Stephane Eranian <eranian@google.com> wrote:
> On Fri, Sep 10, 2010 at 1:41 PM, Peter Zijlstra <peterz@infradead.org> wrote:
>> On Thu, 2010-09-02 at 15:07 -0400, Don Zickus wrote:
>>> Fixes to allow unknown nmis to pass through the perf nmi handler instead
>>> of being swallowed.  Contains patches that are already in Ingo's tree.  Added
>>> here for completeness.  Based on ingo/tip
>>>
>>> Tested on intel/amd
>>>
>>> v2: patch cleanups and consolidation, no code changes
>>>
>>> Don Zickus (1):
>>>   perf, x86: Fix accidentally ack'ing a second event on intel perf
>>>     counter
>>>
>>> Peter Zijlstra (1):
>>>   perf, x86: Fix handle_irq return values
>>>
>>> Robert Richter (1):
>>>   perf, x86: Try to handle unknown nmis with an enabled PMU
>>>
>>>  arch/x86/kernel/cpu/perf_event.c       |   59 +++++++++++++++++++++++++-------
>>>  arch/x86/kernel/cpu/perf_event_intel.c |   15 +++++---
>>>  arch/x86/kernel/cpu/perf_event_p4.c    |    2 +-
>>>  3 files changed, 56 insertions(+), 20 deletions(-)
>>
>> Both Ingo and I are getting Dazed and confused on our AMD machines, it
>> started before yesterday (that is, after backing out all my recent
>> changes it still gets dazed), so I suspect this set.
>>
>> I'll look at getting a trace of the thing, but if any of you has a
>> bright idea...
>
> I still don't buy the back-to-back NMI thing. I suspect there is
> something else going on. I have continued to track it down.
> I got closer yesterday, until I ran into other issues. It may
> have to do with throttling. I am still trying to understanding
> why the OVF_STATUS does not match the check based on
> the counter values.
>
>>
>

Re: [PATCH 0/3 v2] nmi perf fixes

[Don Zickus]

On Fri, Sep 10, 2010 at 01:41:40PM +0200, Peter Zijlstra wrote:
> On Thu, 2010-09-02 at 15:07 -0400, Don Zickus wrote:
> > Fixes to allow unknown nmis to pass through the perf nmi handler instead
> > of being swallowed.  Contains patches that are already in Ingo's tree.  Added
> > here for completeness.  Based on ingo/tip
> > 
> > Tested on intel/amd
> > 
> > v2: patch cleanups and consolidation, no code changes
> > 
> > Don Zickus (1):
> >   perf, x86: Fix accidentally ack'ing a second event on intel perf
> >     counter
> > 
> > Peter Zijlstra (1):
> >   perf, x86: Fix handle_irq return values
> > 
> > Robert Richter (1):
> >   perf, x86: Try to handle unknown nmis with an enabled PMU
> > 
> >  arch/x86/kernel/cpu/perf_event.c       |   59 +++++++++++++++++++++++++-------
> >  arch/x86/kernel/cpu/perf_event_intel.c |   15 +++++---
> >  arch/x86/kernel/cpu/perf_event_p4.c    |    2 +-
> >  3 files changed, 56 insertions(+), 20 deletions(-)
> 
> Both Ingo and I are getting Dazed and confused on our AMD machines, it
> started before yesterday (that is, after backing out all my recent
> changes it still gets dazed), so I suspect this set.
> 
> I'll look at getting a trace of the thing, but if any of you has a
> bright idea...

What are you running to create the problem?  I can try and duplicate it
here.

Cheers,
Don

Re: [PATCH 0/3 v2] nmi perf fixes

[Ingo Molnar]

* Don Zickus <dzickus@redhat.com> wrote:

> On Fri, Sep 10, 2010 at 01:41:40PM +0200, Peter Zijlstra wrote:
> > On Thu, 2010-09-02 at 15:07 -0400, Don Zickus wrote:
> > > Fixes to allow unknown nmis to pass through the perf nmi handler instead
> > > of being swallowed.  Contains patches that are already in Ingo's tree.  Added
> > > here for completeness.  Based on ingo/tip
> > > 
> > > Tested on intel/amd
> > > 
> > > v2: patch cleanups and consolidation, no code changes
> > > 
> > > Don Zickus (1):
> > >   perf, x86: Fix accidentally ack'ing a second event on intel perf
> > >     counter
> > > 
> > > Peter Zijlstra (1):
> > >   perf, x86: Fix handle_irq return values
> > > 
> > > Robert Richter (1):
> > >   perf, x86: Try to handle unknown nmis with an enabled PMU
> > > 
> > >  arch/x86/kernel/cpu/perf_event.c       |   59 +++++++++++++++++++++++++-------
> > >  arch/x86/kernel/cpu/perf_event_intel.c |   15 +++++---
> > >  arch/x86/kernel/cpu/perf_event_p4.c    |    2 +-
> > >  3 files changed, 56 insertions(+), 20 deletions(-)
> > 
> > Both Ingo and I are getting Dazed and confused on our AMD machines, it
> > started before yesterday (that is, after backing out all my recent
> > changes it still gets dazed), so I suspect this set.
> > 
> > I'll look at getting a trace of the thing, but if any of you has a
> > bright idea...
> 
> What are you running to create the problem?  I can try and duplicate 
> it here.

It happens easily here - just running something like:

   perf record -g ./hackbench 10

a couple of times triggers it. Note, unlike with the earlier bug, the 
NMIs are not permanently 'stuck' - and everything continues working. 
Obviously the messages are nasty looking so this is a regression we need 
to fix.

Thanks,

	Ingo

Re: [PATCH 0/3 v2] nmi perf fixes

[Robert Richter]

On 10.09.10 10:46:34, Ingo Molnar wrote:
> 
> * Don Zickus <dzickus@redhat.com> wrote:
> 
> > On Fri, Sep 10, 2010 at 01:41:40PM +0200, Peter Zijlstra wrote:
> > > On Thu, 2010-09-02 at 15:07 -0400, Don Zickus wrote:
> > > > Fixes to allow unknown nmis to pass through the perf nmi handler instead
> > > > of being swallowed.  Contains patches that are already in Ingo's tree.  Added
> > > > here for completeness.  Based on ingo/tip
> > > > 
> > > > Tested on intel/amd
> > > > 
> > > > v2: patch cleanups and consolidation, no code changes
> > > > 
> > > > Don Zickus (1):
> > > >   perf, x86: Fix accidentally ack'ing a second event on intel perf
> > > >     counter
> > > > 
> > > > Peter Zijlstra (1):
> > > >   perf, x86: Fix handle_irq return values
> > > > 
> > > > Robert Richter (1):
> > > >   perf, x86: Try to handle unknown nmis with an enabled PMU
> > > > 
> > > >  arch/x86/kernel/cpu/perf_event.c       |   59 +++++++++++++++++++++++++-------
> > > >  arch/x86/kernel/cpu/perf_event_intel.c |   15 +++++---
> > > >  arch/x86/kernel/cpu/perf_event_p4.c    |    2 +-
> > > >  3 files changed, 56 insertions(+), 20 deletions(-)
> > > 
> > > Both Ingo and I are getting Dazed and confused on our AMD machines, it
> > > started before yesterday (that is, after backing out all my recent
> > > changes it still gets dazed), so I suspect this set.
> > > 
> > > I'll look at getting a trace of the thing, but if any of you has a
> > > bright idea...
> > 
> > What are you running to create the problem?  I can try and duplicate 
> > it here.
> 
> It happens easily here - just running something like:
> 
>    perf record -g ./hackbench 10

I try to reproduce it, which systems are affected?

> 
> a couple of times triggers it. Note, unlike with the earlier bug, the 
> NMIs are not permanently 'stuck' - and everything continues working. 
> Obviously the messages are nasty looking so this is a regression we need 
> to fix.

The patch below adds ratelimits.

-Robert

--

>From 1747710d684302b806b145e5acb590ab2088e5ca Mon Sep 17 00:00:00 2001
From: Robert Richter <robert.richter@amd.com>
Date: Thu, 5 Aug 2010 18:04:31 +0200
Subject: [PATCH] x86: ratelimit NMI messages

In case of a storm of unknown NMIs the cpu get stucked. This patch
adds ratelimits to avoid this.

Signed-off-by: Robert Richter <robert.richter@amd.com>
---
 arch/x86/kernel/traps.c |   16 +++++++++-------
 1 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index 60788de..97a492d 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -366,15 +366,17 @@ unknown_nmi_error(unsigned char reason, struct pt_regs *regs)
 		return;
 	}
 #endif
-	printk(KERN_EMERG
-		"Uhhuh. NMI received for unknown reason %02x on CPU %d.\n",
-			reason, smp_processor_id());
-
-	printk(KERN_EMERG "Do you have a strange power saving mode enabled?\n");
+	if (printk_ratelimit()) {
+		printk(KERN_EMERG "Uhhuh. NMI received for unknown reason"
+		       " %02x on CPU %d.\n", reason, smp_processor_id());
+		printk(KERN_EMERG
+		       "Do you have a strange power saving mode enabled?\n");
+	}
 	if (panic_on_unrecovered_nmi)
 		panic("NMI: Not continuing");
-
-	printk(KERN_EMERG "Dazed and confused, but trying to continue\n");
+	if (printk_ratelimit())
+		printk(KERN_EMERG
+		       "Dazed and confused, but trying to continue\n");
 }
 
 static notrace __kprobes void default_do_nmi(struct pt_regs *regs)
-- 
1.7.1.1



-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH 0/3 v2] nmi perf fixes

[Peter Zijlstra]

On Fri, 2010-09-10 at 17:17 +0200, Robert Richter wrote:
> >    perf record -g ./hackbench 10
> 
> I try to reproduce it, which systems are affected? 

I know for sure Ingo's Barcelona and my Opteron 12xx.

Re: [PATCH 0/3 v2] nmi perf fixes

[Ingo Molnar]

* Peter Zijlstra <peterz@infradead.org> wrote:

> On Fri, 2010-09-10 at 17:17 +0200, Robert Richter wrote:
> > >    perf record -g ./hackbench 10
> > 
> > I try to reproduce it, which systems are affected? 
> 
> I know for sure Ingo's Barcelona and my Opteron 12xx.

Havent seen it on an AMD Athlon64 dual-core box - but havent tried 
really hard there.

Thanks,

	Ingo

Re: [PATCH 0/3 v2] nmi perf fixes

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Peter Zijlstra <peterz@infradead.org> wrote:
> 
> > On Fri, 2010-09-10 at 17:17 +0200, Robert Richter wrote:
> > > >    perf record -g ./hackbench 10
> > > 
> > > I try to reproduce it, which systems are affected? 
> > 
> > I know for sure Ingo's Barcelona and my Opteron 12xx.
> 
> Havent seen it on an AMD Athlon64 dual-core box - but havent tried 
> really hard there.

Also note that i tried _really_ hard to trigger it on an Intel system 
(Nehalem), with multi-hour tests, and it didnt trigger there. So this 
only occurs on AMD systems.

Thanks,

	Ingo

Re: [PATCH 0/3 v2] nmi perf fixes

[Ingo Molnar]

* Robert Richter <robert.richter@amd.com> wrote:

> On 10.09.10 10:46:34, Ingo Molnar wrote:
> > 
> > * Don Zickus <dzickus@redhat.com> wrote:
> > 
> > > On Fri, Sep 10, 2010 at 01:41:40PM +0200, Peter Zijlstra wrote:
> > > > On Thu, 2010-09-02 at 15:07 -0400, Don Zickus wrote:
> > > > > Fixes to allow unknown nmis to pass through the perf nmi handler instead
> > > > > of being swallowed.  Contains patches that are already in Ingo's tree.  Added
> > > > > here for completeness.  Based on ingo/tip
> > > > > 
> > > > > Tested on intel/amd
> > > > > 
> > > > > v2: patch cleanups and consolidation, no code changes
> > > > > 
> > > > > Don Zickus (1):
> > > > >   perf, x86: Fix accidentally ack'ing a second event on intel perf
> > > > >     counter
> > > > > 
> > > > > Peter Zijlstra (1):
> > > > >   perf, x86: Fix handle_irq return values
> > > > > 
> > > > > Robert Richter (1):
> > > > >   perf, x86: Try to handle unknown nmis with an enabled PMU
> > > > > 
> > > > >  arch/x86/kernel/cpu/perf_event.c       |   59 +++++++++++++++++++++++++-------
> > > > >  arch/x86/kernel/cpu/perf_event_intel.c |   15 +++++---
> > > > >  arch/x86/kernel/cpu/perf_event_p4.c    |    2 +-
> > > > >  3 files changed, 56 insertions(+), 20 deletions(-)
> > > > 
> > > > Both Ingo and I are getting Dazed and confused on our AMD machines, it
> > > > started before yesterday (that is, after backing out all my recent
> > > > changes it still gets dazed), so I suspect this set.
> > > > 
> > > > I'll look at getting a trace of the thing, but if any of you has a
> > > > bright idea...
> > > 
> > > What are you running to create the problem?  I can try and duplicate 
> > > it here.
> > 
> > It happens easily here - just running something like:
> > 
> >    perf record -g ./hackbench 10
> 
> I try to reproduce it, which systems are affected?

This is an older Barcelona box, will send you a dump of details off-list.

> > a couple of times triggers it. Note, unlike with the earlier bug, 
> > the NMIs are not permanently 'stuck' - and everything continues 
> > working. Obviously the messages are nasty looking so this is a 
> > regression we need to fix.
> 
> The patch below adds ratelimits.

This doesnt solve things though - even one unnecessary message will 
confuse users.

	Ingo

Re: [PATCH 0/3 v2] nmi perf fixes

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > > It happens easily here - just running something like:
> > > 
> > >    perf record -g ./hackbench 10
> > 
> > I try to reproduce it, which systems are affected?
> 
> This is an older Barcelona box, will send you a dump of details off-list.

Note, i tried to trigger it again after a fresh reboot into tip:88a377e, 
and this time 3 runs of hackbench didnt trigger the message, but the 
first 'perf top' did. YMMV.

	Ingo

[PATCH] x86: fix duplicate calls of the nmi handler

[Robert Richter]

On 10.09.10 10:46:34, Ingo Molnar wrote:
> > > I'll look at getting a trace of the thing, but if any of you has a
> > > bright idea...

I found another patch in my queue, which fixes a duplicate call of the
nmi handler. Since I could not yet reproduce the bug, I am not sure if
this fixes the problem, but it is worth a try.

-Robert

--

>From 037678d4231778c55ed1a19b53d24c7056ae8bbd Mon Sep 17 00:00:00 2001
From: Robert Richter <robert.richter@amd.com>
Date: Fri, 6 Aug 2010 20:45:51 +0200
Subject: [PATCH] x86: fix duplicate calls of the nmi handler

The commit:

 e40b172 x86: Move notify_die from nmi.c to traps.c

moved the nmi handler call to default_do_nmi(). DIE_NMI_IPI and
DIE_NMI are called subsequently now. If the return code is
!NOTIFY_STOP, then the handlers are called twice. This patch fixes
this.

Signed-off-by: Robert Richter <robert.richter@amd.com>
---
 arch/x86/kernel/apic/hw_nmi.c    |    1 -
 arch/x86/kernel/cpu/perf_event.c |    1 -
 arch/x86/oprofile/nmi_int.c      |    1 -
 3 files changed, 0 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kernel/apic/hw_nmi.c b/arch/x86/kernel/apic/hw_nmi.c
index cefd694..61a3ad7 100644
--- a/arch/x86/kernel/apic/hw_nmi.c
+++ b/arch/x86/kernel/apic/hw_nmi.c
@@ -52,7 +52,6 @@ arch_trigger_all_cpu_backtrace_handler(struct notifier_block *self,
 	int cpu = smp_processor_id();
 
 	switch (cmd) {
-	case DIE_NMI:
 	case DIE_NMI_IPI:
 		break;
 
diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index 3efdf28..87dc9e2 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1219,7 +1219,6 @@ perf_event_nmi_handler(struct notifier_block *self,
 		return NOTIFY_DONE;
 
 	switch (cmd) {
-	case DIE_NMI:
 	case DIE_NMI_IPI:
 		break;
 	case DIE_NMIUNKNOWN:
diff --git a/arch/x86/oprofile/nmi_int.c b/arch/x86/oprofile/nmi_int.c
index cfe4faa..e0132bf 100644
--- a/arch/x86/oprofile/nmi_int.c
+++ b/arch/x86/oprofile/nmi_int.c
@@ -64,7 +64,6 @@ static int profile_exceptions_notify(struct notifier_block *self,
 	int ret = NOTIFY_DONE;
 
 	switch (val) {
-	case DIE_NMI:
 	case DIE_NMI_IPI:
 		if (ctr_running)
 			model->check_ctrs(args->regs, &__get_cpu_var(cpu_msrs));
-- 
1.7.1.1



> > 
> > What are you running to create the problem?  I can try and duplicate 
> > it here.
> 
> It happens easily here - just running something like:
> 
>    perf record -g ./hackbench 10
> 
> a couple of times triggers it. Note, unlike with the earlier bug, the 
> NMIs are not permanently 'stuck' - and everything continues working. 
> Obviously the messages are nasty looking so this is a regression we need 
> to fix.
> 
> Thanks,
> 
> 	Ingo
> 

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Peter Zijlstra]

On Fri, 2010-09-10 at 17:56 +0200, Robert Richter wrote:
> On 10.09.10 10:46:34, Ingo Molnar wrote:
> > > > I'll look at getting a trace of the thing, but if any of you has
> a
> > > > bright idea...
> 
> I found another patch in my queue, which fixes a duplicate call of the
> nmi handler. Since I could not yet reproduce the bug, I am not sure if
> this fixes the problem, but it is worth a try.
> 
> 

           <...>-6114  [001]   340.053060: perf_event_nmi_handler: NMI-handled(1): 34833 27411 1
           <...>-6114  [001]   340.053061: perf_event_nmi_handler: NMI-stop: 34833 27411 1
           <...>-6114  [001]   340.053062: perf_event_nmi_handler: NMI: 34834 27411 1
           <...>-6114  [001]   340.053063: x86_pmu_handle_irq: OVERFLOW: 1
           <...>-6114  [001]   340.053064: x86_pmu_handle_irq: HANDLED: 1
           <...>-6114  [001]   340.053067: perf_event_nmi_handler: NMI-handled(1): 34834 27411 1
           <...>-6114  [001]   340.053068: perf_event_nmi_handler: NMI-stop: 34834 27411 1
           <...>-6114  [001]   340.053069: perf_event_nmi_handler: NMI: 34835 27411 1
           <...>-6114  [001]   340.053070: perf_event_nmi_handler: NMI-handled(0): 34835 27411 1
           <...>-6114  [001]   340.053071: perf_event_nmi_handler: NMI: 34835 27411 1
           <...>-6114  [001]   340.053072: perf_event_nmi_handler: NMI: 34835 27411 1
           <...>-6114  [001]   340.053073: perf_event_nmi_handler: NMI-fail


Nope.. it doesn't get tagged because handled == 1, not > 1.

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Ingo Molnar]

* Robert Richter <robert.richter@amd.com> wrote:

> On 10.09.10 10:46:34, Ingo Molnar wrote:
> > > > I'll look at getting a trace of the thing, but if any of you has a
> > > > bright idea...
> 
> I found another patch in my queue, which fixes a duplicate call of the 
> nmi handler. Since I could not yet reproduce the bug, I am not sure if 
> this fixes the problem, but it is worth a try.

Robert, can you reproduce the problem? It triggers rather easily on both 
Peter's and my AMD systems.

	Ingo

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Robert Richter]

On 11.09.10 05:41:57, Ingo Molnar wrote:
> Robert, can you reproduce the problem? It triggers rather easily on both 
> Peter's and my AMD systems.

I was testing 88a377e and so far it didn't trigger on the system I
have available, will try other systems too. Can you send me your
config too?

Thanks,

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 383 bytes --]


* Robert Richter <robert.richter@amd.com> wrote:

> On 11.09.10 05:41:57, Ingo Molnar wrote:
> > Robert, can you reproduce the problem? It triggers rather easily on both 
> > Peter's and my AMD systems.
> 
> I was testing 88a377e and so far it didn't trigger on the system I 
> have available, will try other systems too. Can you send me your 
> config too?

Sure, attached.

	Ingo

[-- Attachment #2: config --]
[-- Type: text/plain, Size: 83494 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.36-rc3
# Sat Sep 11 15:05:05 2010
#
CONFIG_64BIT=y
# CONFIG_X86_32 is not set
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11"
# CONFIG_KTIME_SCALAR is not set
CONFIG_ARCH_CPU_PROBE_RELEASE=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_CONSTRUCTORS=y

#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_LZO is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
# CONFIG_TASK_XACCT is not set
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=64
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_IKCONFIG is not set
CONFIG_LOG_BUF_SHIFT=20
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
# CONFIG_CGROUPS is not set
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y
CONFIG_RELAY=y
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_IPC_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_NET_NS is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_LZO=y
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_EVENTS=y

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
CONFIG_PERF_COUNTERS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_COMPAT_BRK=y
CONFIG_SLAB=y
# CONFIG_SLUB is not set
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
CONFIG_OPROFILE=m
# CONFIG_OPROFILE_EVENT_MULTIPLEX is not set
CONFIG_HAVE_OPROFILE=y
CONFIG_KPROBES=y
CONFIG_OPTPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_KRETPROBES=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y

#
# GCOV-based kernel profiling
#
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_MODVERSIONS=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
# CONFIG_BLK_DEV_BSG is not set
# CONFIG_BLK_DEV_INTEGRITY is not set
CONFIG_BLOCK_COMPAT=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_INLINE_SPIN_TRYLOCK is not set
# CONFIG_INLINE_SPIN_TRYLOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK is not set
# CONFIG_INLINE_SPIN_LOCK_BH is not set
# CONFIG_INLINE_SPIN_LOCK_IRQ is not set
# CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set
# CONFIG_INLINE_SPIN_UNLOCK is not set
# CONFIG_INLINE_SPIN_UNLOCK_BH is not set
# CONFIG_INLINE_SPIN_UNLOCK_IRQ is not set
# CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_READ_TRYLOCK is not set
# CONFIG_INLINE_READ_LOCK is not set
# CONFIG_INLINE_READ_LOCK_BH is not set
# CONFIG_INLINE_READ_LOCK_IRQ is not set
# CONFIG_INLINE_READ_LOCK_IRQSAVE is not set
# CONFIG_INLINE_READ_UNLOCK is not set
# CONFIG_INLINE_READ_UNLOCK_BH is not set
# CONFIG_INLINE_READ_UNLOCK_IRQ is not set
# CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set
# CONFIG_INLINE_WRITE_TRYLOCK is not set
# CONFIG_INLINE_WRITE_LOCK is not set
# CONFIG_INLINE_WRITE_LOCK_BH is not set
# CONFIG_INLINE_WRITE_LOCK_IRQ is not set
# CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set
# CONFIG_INLINE_WRITE_UNLOCK is not set
# CONFIG_INLINE_WRITE_UNLOCK_BH is not set
# CONFIG_INLINE_WRITE_UNLOCK_IRQ is not set
# CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set
# CONFIG_MUTEX_SPIN_ON_OWNER is not set
CONFIG_FREEZER=y

#
# Processor type and features
#
# CONFIG_NO_HZ is not set
# CONFIG_HIGH_RES_TIMERS is not set
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
# CONFIG_SPARSE_IRQ is not set
CONFIG_X86_MPPARSE=y
CONFIG_X86_EXTENDED_PLATFORM=y
# CONFIG_X86_VSMP is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
# CONFIG_PARAVIRT_GUEST is not set
CONFIG_NO_BOOTMEM=y
# CONFIG_MEMTEST is not set
# CONFIG_MK8 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_GENERIC_CPU=y
CONFIG_X86_CPU=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=7
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
# CONFIG_AMD_IOMMU is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
# CONFIG_IOMMU_API is not set
# CONFIG_MAXSMP is not set
CONFIG_NR_CPUS=255
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
# CONFIG_X86_MCE_INJECT is not set
CONFIG_X86_THERMAL_VECTOR=y
# CONFIG_I8K is not set
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
# CONFIG_MICROCODE_AMD is not set
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
CONFIG_NUMA=y
CONFIG_K8_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NODES_SPAN_OTHER_NODES=y
# CONFIG_NUMA_EMU is not set
CONFIG_NODES_SHIFT=6
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_NEED_MULTIPLE_NODES=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
# CONFIG_MEMORY_HOTPLUG is not set
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=999999
# CONFIG_COMPACTION is not set
CONFIG_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
# CONFIG_KSM is not set
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
# CONFIG_MEMORY_FAILURE is not set
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
# CONFIG_MTRR_SANITIZER is not set
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
# CONFIG_EFI is not set
# CONFIG_SECCOMP is not set
# CONFIG_CC_STACKPROTECTOR is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
# CONFIG_SCHED_HRTICK is not set
CONFIG_KEXEC=y
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x1000000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
CONFIG_COMPAT_VDSO=y
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_HAVE_ARCH_EARLY_PFN_TO_NID=y
CONFIG_USE_PERCPU_NUMA_NODE_ID=y

#
# Power management and ACPI options
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_SLEEP=y
CONFIG_SUSPEND_NVS=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
# CONFIG_HIBERNATION is not set
# CONFIG_PM_RUNTIME is not set
CONFIG_PM_OPS=y
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
# CONFIG_ACPI_POWER_METER is not set
CONFIG_ACPI_SYSFS_POWER=y
# CONFIG_ACPI_EC_DEBUGFS is not set
CONFIG_ACPI_PROC_EVENT=y
CONFIG_ACPI_AC=m
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=m
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
# CONFIG_ACPI_PROCESSOR_AGGREGATOR is not set
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_NUMA=y
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
# CONFIG_ACPI_PCI_SLOT is not set
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_SBS=m
# CONFIG_ACPI_HED is not set
# CONFIG_ACPI_APEI is not set
# CONFIG_SFI is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_DEBUG=y
CONFIG_CPU_FREQ_STAT=m
CONFIG_CPU_FREQ_STAT_DETAILS=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=m
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=m
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m

#
# CPUFreq processor drivers
#
# CONFIG_X86_PCC_CPUFREQ is not set
CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_X86_POWERNOW_K8=y
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
# CONFIG_X86_P4_CLOCKMOD is not set

#
# shared options
#
# CONFIG_X86_SPEEDSTEP_LIB is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
# CONFIG_INTEL_IDLE is not set

#
# Memory power savings
#
# CONFIG_I7300_IDLE is not set

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCI_CNB20LE_QUIRK is not set
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=m
CONFIG_PCIEAER=y
# CONFIG_PCIE_ECRC is not set
# CONFIG_PCIEAER_INJECT is not set
CONFIG_PCIEASPM=y
# CONFIG_PCIEASPM_DEBUG is not set
CONFIG_ARCH_SUPPORTS_MSI=y
# CONFIG_PCI_MSI is not set
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_STUB is not set
CONFIG_HT_IRQ=y
# CONFIG_PCI_IOV is not set
CONFIG_PCI_IOAPIC=y
CONFIG_ISA_DMA_API=y
CONFIG_K8_NB=y
CONFIG_PCCARD=y
CONFIG_PCMCIA=y
CONFIG_PCMCIA_LOAD_CIS=y
CONFIG_CARDBUS=y

#
# PC-card bridges
#
CONFIG_YENTA=y
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PD6729=m
CONFIG_I82092=m
CONFIG_PCCARD_NONSTATIC=y
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_ACPI=m
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
# CONFIG_HOTPLUG_PCI_CPCI is not set
CONFIG_HOTPLUG_PCI_SHPC=m

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
# CONFIG_HAVE_AOUT is not set
CONFIG_BINFMT_MISC=y
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=y
# CONFIG_XFRM_SUB_POLICY is not set
CONFIG_XFRM_MIGRATE=y
# CONFIG_XFRM_STATISTICS is not set
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
CONFIG_NET_KEY_MIGRATE=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
# CONFIG_IP_MROUTE_MULTIPLE_TABLES is not set
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=y
CONFIG_TCP_CONG_CUBIC=m
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
# CONFIG_TCP_CONG_YEAH is not set
# CONFIG_TCP_CONG_ILLINOIS is not set
CONFIG_DEFAULT_BIC=y
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="bic"
CONFIG_TCP_MD5SIG=y
CONFIG_IPV6=m
CONFIG_IPV6_PRIVACY=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
# CONFIG_IPV6_OPTIMISTIC_DAD is not set
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
CONFIG_INET6_IPCOMP=m
# CONFIG_IPV6_MIP6 is not set
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=m
CONFIG_IPV6_SIT=m
# CONFIG_IPV6_SIT_6RD is not set
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
# CONFIG_IPV6_MULTIPLE_TABLES is not set
# CONFIG_IPV6_MROUTE is not set
CONFIG_NETLABEL=y
CONFIG_NETWORK_SECMARK=y
# CONFIG_NETWORK_PHY_TIMESTAMPING is not set
CONFIG_NETFILTER=y
CONFIG_NETFILTER_DEBUG=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=y

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=y
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CT_PROTO_DCCP=m
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_CT_PROTO_SCTP=m
# CONFIG_NF_CT_PROTO_UDPLITE is not set
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
# CONFIG_NF_CT_NETLINK is not set
# CONFIG_NETFILTER_TPROXY is not set
CONFIG_NETFILTER_XTABLES=m

#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m

#
# Xtables targets
#
# CONFIG_NETFILTER_XT_TARGET_CHECKSUM is not set
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
# CONFIG_NETFILTER_XT_TARGET_CT is not set
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
# CONFIG_NETFILTER_XT_TARGET_IDLETIMER is not set
# CONFIG_NETFILTER_XT_TARGET_LED is not set
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
# CONFIG_NETFILTER_XT_TARGET_RATEEST is not set
# CONFIG_NETFILTER_XT_TARGET_TEE is not set
# CONFIG_NETFILTER_XT_TARGET_TRACE is not set
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
# CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP is not set

#
# Xtables matches
#
# CONFIG_NETFILTER_XT_MATCH_CLUSTER is not set
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
# CONFIG_NETFILTER_XT_MATCH_CONNLIMIT is not set
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
# CONFIG_NETFILTER_XT_MATCH_CPU is not set
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
# CONFIG_NETFILTER_XT_MATCH_IPRANGE is not set
# CONFIG_NETFILTER_XT_MATCH_IPVS is not set
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
# CONFIG_NETFILTER_XT_MATCH_OSF is not set
# CONFIG_NETFILTER_XT_MATCH_OWNER is not set
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
# CONFIG_NETFILTER_XT_MATCH_RATEEST is not set
CONFIG_NETFILTER_XT_MATCH_REALM=m
# CONFIG_NETFILTER_XT_MATCH_RECENT is not set
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
# CONFIG_NETFILTER_XT_MATCH_TIME is not set
# CONFIG_NETFILTER_XT_MATCH_U32 is not set
CONFIG_IP_VS=m
# CONFIG_IP_VS_IPV6 is not set
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
# CONFIG_IP_VS_PROTO_SCTP is not set

#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m

#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PROTO_DCCP=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PROTO_SCTP=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_NF_NAT_SIP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
# CONFIG_IP_NF_SECURITY is not set
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

#
# IPv6: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV6=m
CONFIG_IP6_NF_QUEUE=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_AH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_MH=m
CONFIG_IP6_NF_MATCH_RT=m
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_TARGET_LOG=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
# CONFIG_IP6_NF_SECURITY is not set

#
# DECnet: Netfilter Configuration
#
# CONFIG_DECNET_NF_GRABULATOR is not set
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
# CONFIG_BRIDGE_EBT_IP6 is not set
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_ULOG=m
# CONFIG_BRIDGE_EBT_NFLOG is not set
CONFIG_IP_DCCP=m
CONFIG_INET_DCCP_DIAG=m

#
# DCCP CCIDs Configuration (EXPERIMENTAL)
#
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
CONFIG_IP_DCCP_CCID3=y
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_CCID3_RTO=100
CONFIG_IP_DCCP_TFRC_LIB=y

#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
# CONFIG_NET_DCCPPROBE is not set
CONFIG_IP_SCTP=m
# CONFIG_NET_SCTPPROBE is not set
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_HMAC_NONE is not set
# CONFIG_SCTP_HMAC_SHA1 is not set
CONFIG_SCTP_HMAC_MD5=y
# CONFIG_RDS is not set
CONFIG_TIPC=m
# CONFIG_TIPC_ADVANCED is not set
# CONFIG_TIPC_DEBUG is not set
CONFIG_ATM=m
CONFIG_ATM_CLIP=m
# CONFIG_ATM_CLIP_NO_ICMP is not set
CONFIG_ATM_LANE=m
# CONFIG_ATM_MPOA is not set
CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set
# CONFIG_L2TP is not set
CONFIG_STP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
# CONFIG_NET_DSA is not set
CONFIG_VLAN_8021Q=m
# CONFIG_VLAN_8021Q_GVRP is not set
CONFIG_DECNET=m
CONFIG_DECNET_ROUTER=y
CONFIG_LLC=y
# CONFIG_LLC2 is not set
CONFIG_IPX=m
# CONFIG_IPX_INTERN is not set
CONFIG_ATALK=m
CONFIG_DEV_APPLETALK=m
CONFIG_IPDDP=m
CONFIG_IPDDP_ENCAP=y
CONFIG_IPDDP_DECAP=y
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
CONFIG_WAN_ROUTER=m
# CONFIG_PHONET is not set
# CONFIG_IEEE802154 is not set
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=m
# CONFIG_NET_SCH_MULTIQ is not set
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
# CONFIG_NET_SCH_DRR is not set
CONFIG_NET_SCH_INGRESS=m

#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
# CONFIG_NET_CLS_FLOW is not set
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
# CONFIG_NET_ACT_NAT is not set
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
# CONFIG_NET_ACT_SKBEDIT is not set
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
# CONFIG_DCB is not set
CONFIG_DNS_RESOLVER=y
CONFIG_RPS=y

#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_NET_TCPPROBE is not set
# CONFIG_NET_DROP_MONITOR is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
CONFIG_IRDA=m

#
# IrDA protocols
#
CONFIG_IRLAN=m
CONFIG_IRNET=m
CONFIG_IRCOMM=m
# CONFIG_IRDA_ULTRA is not set

#
# IrDA options
#
CONFIG_IRDA_CACHE_LAST_LSAP=y
CONFIG_IRDA_FAST_RR=y
# CONFIG_IRDA_DEBUG is not set

#
# Infrared-port device drivers
#

#
# SIR device drivers
#
CONFIG_IRTTY_SIR=m

#
# Dongle support
#
CONFIG_DONGLE=y
CONFIG_ESI_DONGLE=m
CONFIG_ACTISYS_DONGLE=m
CONFIG_TEKRAM_DONGLE=m
CONFIG_TOIM3232_DONGLE=m
CONFIG_LITELINK_DONGLE=m
CONFIG_MA600_DONGLE=m
CONFIG_GIRBIL_DONGLE=m
CONFIG_MCP2120_DONGLE=m
CONFIG_OLD_BELKIN_DONGLE=m
CONFIG_ACT200L_DONGLE=m
# CONFIG_KINGSUN_DONGLE is not set
# CONFIG_KSDAZZLE_DONGLE is not set
# CONFIG_KS959_DONGLE is not set

#
# FIR device drivers
#
CONFIG_USB_IRDA=m
CONFIG_SIGMATEL_FIR=m
CONFIG_NSC_FIR=m
CONFIG_WINBOND_FIR=m
CONFIG_SMC_IRCC_FIR=m
CONFIG_ALI_FIR=m
CONFIG_VLSI_FIR=m
CONFIG_VIA_FIR=m
CONFIG_MCS_FIR=m
CONFIG_BT=m
CONFIG_BT_L2CAP=m
CONFIG_BT_SCO=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_HIDP=m

#
# Bluetooth device drivers
#
# CONFIG_BT_HCIBTUSB is not set
# CONFIG_BT_HCIBTSDIO is not set
CONFIG_BT_HCIUART=m
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
# CONFIG_BT_HCIUART_ATH3K is not set
# CONFIG_BT_HCIUART_LL is not set
CONFIG_BT_HCIBCM203X=m
CONFIG_BT_HCIBPA10X=m
CONFIG_BT_HCIBFUSB=m
CONFIG_BT_HCIDTL1=m
CONFIG_BT_HCIBT3C=m
CONFIG_BT_HCIBLUECARD=m
CONFIG_BT_HCIBTUART=m
CONFIG_BT_HCIVHCI=m
# CONFIG_BT_MRVL is not set
# CONFIG_AF_RXRPC is not set
CONFIG_FIB_RULES=y
CONFIG_WIRELESS=y
# CONFIG_CFG80211 is not set
# CONFIG_LIB80211 is not set

#
# CFG80211 needs to be enabled for MAC80211
#

#
# Some wireless drivers require a rate control algorithm
#
# CONFIG_WIMAX is not set
CONFIG_RFKILL=m
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
# CONFIG_CAIF is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
# CONFIG_DEVTMPFS is not set
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
# CONFIG_MTD is not set
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_SERIAL=m
# CONFIG_PARPORT_PC_FIFO is not set
# CONFIG_PARPORT_PC_SUPERIO is not set
CONFIG_PARPORT_PC_PCMCIA=m
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_AX88796 is not set
CONFIG_PARPORT_1284=y
CONFIG_PARPORT_NOT_PC=y
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=m
# CONFIG_PARIDE is not set
CONFIG_BLK_CPQ_DA=y
CONFIG_BLK_CPQ_CISS_DA=m
CONFIG_CISS_SCSI_TAPE=y
CONFIG_BLK_DEV_DAC960=m
CONFIG_BLK_DEV_UMEM=m
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
# CONFIG_BLK_DEV_DRBD is not set
CONFIG_BLK_DEV_NBD=m
CONFIG_BLK_DEV_SX8=m
CONFIG_BLK_DEV_UB=m
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=16384
# CONFIG_BLK_DEV_XIP is not set
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
CONFIG_ATA_OVER_ETH=m
# CONFIG_BLK_DEV_HD is not set
CONFIG_MISC_DEVICES=y
# CONFIG_AD525X_DPOT is not set
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
CONFIG_SGI_IOC4=m
CONFIG_TIFM_CORE=m
CONFIG_TIFM_7XX1=m
# CONFIG_ICS932S401 is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_CS5535_MFGPT is not set
# CONFIG_HP_ILO is not set
# CONFIG_ISL29003 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_SENSORS_BH1780 is not set
# CONFIG_HMC6352 is not set
# CONFIG_DS1682 is not set
# CONFIG_VMWARE_BALLOON is not set
# CONFIG_BMP085 is not set
# CONFIG_C2PORT is not set

#
# EEPROM support
#
# CONFIG_EEPROM_AT24 is not set
# CONFIG_EEPROM_LEGACY is not set
# CONFIG_EEPROM_MAX6875 is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_CB710_CORE is not set
# CONFIG_IWMC3200TOP is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_TGT=m
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=m
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_FC_ATTRS=m
# CONFIG_SCSI_FC_TGT_ATTRS is not set
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=m
# CONFIG_ISCSI_BOOT_SYSFS is not set
# CONFIG_SCSI_CXGB3_ISCSI is not set
# CONFIG_SCSI_BNX2_ISCSI is not set
# CONFIG_BE2ISCSI is not set
CONFIG_BLK_DEV_3W_XXXX_RAID=m
# CONFIG_SCSI_HPSA is not set
CONFIG_SCSI_3W_9XXX=m
# CONFIG_SCSI_3W_SAS is not set
CONFIG_SCSI_ACARD=m
CONFIG_SCSI_AACRAID=m
CONFIG_SCSI_AIC7XXX=y
CONFIG_AIC7XXX_CMDS_PER_DEVICE=4
CONFIG_AIC7XXX_RESET_DELAY_MS=15000
# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
CONFIG_AIC7XXX_DEBUG_MASK=0
# CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set
CONFIG_SCSI_AIC7XXX_OLD=m
CONFIG_SCSI_AIC79XX=m
CONFIG_AIC79XX_CMDS_PER_DEVICE=4
CONFIG_AIC79XX_RESET_DELAY_MS=15000
# CONFIG_AIC79XX_DEBUG_ENABLE is not set
CONFIG_AIC79XX_DEBUG_MASK=0
# CONFIG_AIC79XX_REG_PRETTY_PRINT is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_MVSAS is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
CONFIG_SCSI_ARCMSR=m
# CONFIG_SCSI_ARCMSR_AER is not set
CONFIG_MEGARAID_NEWGEN=y
CONFIG_MEGARAID_MM=m
CONFIG_MEGARAID_MAILBOX=m
CONFIG_MEGARAID_LEGACY=m
CONFIG_MEGARAID_SAS=m
# CONFIG_SCSI_MPT2SAS is not set
CONFIG_SCSI_HPTIOP=m
CONFIG_SCSI_BUSLOGIC=m
# CONFIG_VMWARE_PVSCSI is not set
# CONFIG_LIBFC is not set
# CONFIG_LIBFCOE is not set
# CONFIG_FCOE is not set
# CONFIG_FCOE_FNIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
CONFIG_SCSI_GDTH=m
CONFIG_SCSI_IPS=m
CONFIG_SCSI_INITIO=m
CONFIG_SCSI_INIA100=m
CONFIG_SCSI_PPA=m
CONFIG_SCSI_IMM=m
# CONFIG_SCSI_IZIP_EPP16 is not set
# CONFIG_SCSI_IZIP_SLOW_CTR is not set
CONFIG_SCSI_STEX=m
CONFIG_SCSI_SYM53C8XX_2=m
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
CONFIG_SCSI_SYM53C8XX_MMIO=y
# CONFIG_SCSI_IPR is not set
CONFIG_SCSI_QLOGIC_1280=m
CONFIG_SCSI_QLA_FC=m
CONFIG_SCSI_QLA_ISCSI=m
CONFIG_SCSI_LPFC=m
# CONFIG_SCSI_LPFC_DEBUG_FS is not set
CONFIG_SCSI_DC395x=m
CONFIG_SCSI_DC390T=m
# CONFIG_SCSI_PMCRAID is not set
# CONFIG_SCSI_PM8001 is not set
CONFIG_SCSI_SRP=m
# CONFIG_SCSI_BFA_FC is not set
# CONFIG_SCSI_LOWLEVEL_PCMCIA is not set
# CONFIG_SCSI_DH is not set
# CONFIG_SCSI_OSD_INITIATOR is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y

#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=y
# CONFIG_SATA_AHCI_PLATFORM is not set
CONFIG_SATA_INIC162X=m
CONFIG_SATA_SIL24=m
CONFIG_ATA_SFF=y

#
# SFF controllers with custom DMA interface
#
CONFIG_PDC_ADMA=m
CONFIG_SATA_QSTOR=m
CONFIG_SATA_SX4=m
CONFIG_ATA_BMDMA=y

#
# SATA SFF controllers with BMDMA
#
CONFIG_ATA_PIIX=y
CONFIG_SATA_MV=m
CONFIG_SATA_NV=y
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SIL=m
CONFIG_SATA_SIS=m
CONFIG_SATA_SVW=m
CONFIG_SATA_ULI=m
CONFIG_SATA_VIA=m
CONFIG_SATA_VITESSE=m

#
# PATA SFF controllers with BMDMA
#
CONFIG_PATA_ALI=m
CONFIG_PATA_AMD=y
CONFIG_PATA_ARTOP=m
CONFIG_PATA_ATIIXP=m
# CONFIG_PATA_ATP867X is not set
CONFIG_PATA_CMD64X=m
CONFIG_PATA_CS5520=m
CONFIG_PATA_CS5530=m
CONFIG_PATA_CYPRESS=m
CONFIG_PATA_EFAR=m
CONFIG_PATA_HPT366=m
CONFIG_PATA_HPT37X=m
CONFIG_PATA_HPT3X2N=m
CONFIG_PATA_HPT3X3=m
# CONFIG_PATA_HPT3X3_DMA is not set
CONFIG_PATA_IT8213=m
CONFIG_PATA_IT821X=m
CONFIG_PATA_JMICRON=m
CONFIG_PATA_MARVELL=m
CONFIG_PATA_NETCELL=m
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87415 is not set
CONFIG_PATA_OLDPIIX=y
CONFIG_PATA_OPTIDMA=m
CONFIG_PATA_PDC2027X=m
CONFIG_PATA_PDC_OLD=m
CONFIG_PATA_RADISYS=m
# CONFIG_PATA_RDC is not set
CONFIG_PATA_SC1200=m
# CONFIG_PATA_SCH is not set
CONFIG_PATA_SERVERWORKS=m
CONFIG_PATA_SIL680=m
CONFIG_PATA_SIS=m
# CONFIG_PATA_TOSHIBA is not set
CONFIG_PATA_TRIFLEX=m
CONFIG_PATA_VIA=m
CONFIG_PATA_WINBOND=m

#
# PIO-only SFF controllers
#
# CONFIG_PATA_CMD640_PCI is not set
CONFIG_PATA_MPIIX=m
CONFIG_PATA_NS87410=m
CONFIG_PATA_OPTI=m
CONFIG_PATA_PCMCIA=m
CONFIG_PATA_RZ1000=m

#
# Generic fallback / legacy drivers
#
# CONFIG_PATA_ACPI is not set
CONFIG_ATA_GENERIC=m
# CONFIG_PATA_LEGACY is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
# CONFIG_MULTICORE_RAID456 is not set
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
CONFIG_BLK_DEV_DM=m
# CONFIG_DM_DEBUG is not set
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MIRROR=m
# CONFIG_DM_LOG_USERSPACE is not set
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
# CONFIG_DM_MULTIPATH_QL is not set
# CONFIG_DM_MULTIPATH_ST is not set
# CONFIG_DM_DELAY is not set
# CONFIG_DM_UEVENT is not set
CONFIG_FUSION=y
CONFIG_FUSION_SPI=m
CONFIG_FUSION_FC=m
CONFIG_FUSION_SAS=m
CONFIG_FUSION_MAX_SGE=40
CONFIG_FUSION_CTL=m
CONFIG_FUSION_LAN=m
# CONFIG_FUSION_LOGGING is not set

#
# IEEE 1394 (FireWire) support
#

#
# You can enable one or both FireWire driver stacks.
#

#
# The newer stack is recommended.
#
# CONFIG_FIREWIRE is not set
CONFIG_IEEE1394=m
CONFIG_IEEE1394_OHCI1394=m
CONFIG_IEEE1394_PCILYNX=m
CONFIG_IEEE1394_SBP2=m
# CONFIG_IEEE1394_SBP2_PHYS_DMA is not set
CONFIG_IEEE1394_ETH1394_ROM_ENTRY=y
CONFIG_IEEE1394_ETH1394=m
CONFIG_IEEE1394_RAWIO=m
CONFIG_IEEE1394_VIDEO1394=m
CONFIG_IEEE1394_DV1394=m
# CONFIG_IEEE1394_VERBOSEDEBUG is not set
# CONFIG_FIREWIRE_NOSY is not set
CONFIG_I2O=m
# CONFIG_I2O_LCT_NOTIFY_ON_CHANGES is not set
CONFIG_I2O_EXT_ADAPTEC=y
CONFIG_I2O_EXT_ADAPTEC_DMA64=y
# CONFIG_I2O_CONFIG is not set
CONFIG_I2O_BUS=m
CONFIG_I2O_BLOCK=m
CONFIG_I2O_SCSI=m
CONFIG_I2O_PROC=m
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_IFB=m
CONFIG_DUMMY=m
CONFIG_BONDING=m
# CONFIG_MACVLAN is not set
CONFIG_EQUALIZER=m
CONFIG_TUN=m
# CONFIG_VETH is not set
CONFIG_NET_SB1000=m
# CONFIG_ARCNET is not set
CONFIG_PHYLIB=y

#
# MII PHY device drivers
#
CONFIG_MARVELL_PHY=m
CONFIG_DAVICOM_PHY=m
CONFIG_QSEMI_PHY=m
CONFIG_LXT_PHY=m
CONFIG_CICADA_PHY=m
CONFIG_VITESSE_PHY=m
CONFIG_SMSC_PHY=m
CONFIG_BROADCOM_PHY=m
# CONFIG_ICPLUS_PHY is not set
# CONFIG_REALTEK_PHY is not set
# CONFIG_NATIONAL_PHY is not set
# CONFIG_STE10XP is not set
# CONFIG_LSI_ET1011C_PHY is not set
# CONFIG_MICREL_PHY is not set
# CONFIG_FIXED_PHY is not set
# CONFIG_MDIO_BITBANG is not set
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
CONFIG_HAPPYMEAL=m
CONFIG_SUNGEM=m
CONFIG_CASSINI=m
CONFIG_NET_VENDOR_3COM=y
CONFIG_VORTEX=y
CONFIG_TYPHOON=m
# CONFIG_ETHOC is not set
# CONFIG_DNET is not set
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_DE2104X_DSL=0
CONFIG_TULIP=m
# CONFIG_TULIP_MWI is not set
CONFIG_TULIP_MMIO=y
# CONFIG_TULIP_NAPI is not set
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
CONFIG_ULI526X=m
CONFIG_PCMCIA_XIRCOM=m
# CONFIG_HP100 is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
# CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set
# CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set
# CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set
CONFIG_NET_PCI=y
CONFIG_PCNET32=m
CONFIG_AMD8111_ETH=m
CONFIG_ADAPTEC_STARFIRE=m
# CONFIG_KSZ884X_PCI is not set
CONFIG_B44=m
CONFIG_B44_PCI_AUTOSELECT=y
CONFIG_B44_PCICORE_AUTOSELECT=y
CONFIG_B44_PCI=y
CONFIG_FORCEDETH=y
CONFIG_E100=y
CONFIG_FEALNX=m
CONFIG_NATSEMI=m
CONFIG_NE2K_PCI=m
CONFIG_8139CP=m
CONFIG_8139TOO=y
# CONFIG_8139TOO_PIO is not set
# CONFIG_8139TOO_TUNE_TWISTER is not set
CONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_R6040 is not set
CONFIG_SIS900=m
CONFIG_EPIC100=m
# CONFIG_SMSC9420 is not set
CONFIG_SUNDANCE=m
# CONFIG_SUNDANCE_MMIO is not set
# CONFIG_TLAN is not set
# CONFIG_KS8851_MLL is not set
CONFIG_VIA_RHINE=m
CONFIG_VIA_RHINE_MMIO=y
CONFIG_SC92031=m
CONFIG_NET_POCKET=y
CONFIG_ATP=m
CONFIG_DE600=m
CONFIG_DE620=m
# CONFIG_ATL2 is not set
CONFIG_NETDEV_1000=y
CONFIG_ACENIC=m
# CONFIG_ACENIC_OMIT_TIGON_I is not set
CONFIG_DL2K=m
CONFIG_E1000=y
CONFIG_E1000E=y
# CONFIG_IP1000 is not set
# CONFIG_IGB is not set
# CONFIG_IGBVF is not set
CONFIG_NS83820=m
CONFIG_HAMACHI=m
CONFIG_YELLOWFIN=m
CONFIG_R8169=m
CONFIG_R8169_VLAN=y
# CONFIG_SIS190 is not set
CONFIG_SKGE=m
# CONFIG_SKGE_DEBUG is not set
CONFIG_SKY2=m
# CONFIG_SKY2_DEBUG is not set
CONFIG_VIA_VELOCITY=m
CONFIG_TIGON3=y
CONFIG_BNX2=m
# CONFIG_CNIC is not set
CONFIG_QLA3XXX=m
CONFIG_ATL1=m
# CONFIG_ATL1E is not set
# CONFIG_ATL1C is not set
# CONFIG_JME is not set
CONFIG_NETDEV_10000=y
CONFIG_MDIO=m
CONFIG_CHELSIO_T1=m
CONFIG_CHELSIO_T1_1G=y
CONFIG_CHELSIO_T3_DEPENDS=y
CONFIG_CHELSIO_T3=m
CONFIG_CHELSIO_T4_DEPENDS=y
# CONFIG_CHELSIO_T4 is not set
CONFIG_CHELSIO_T4VF_DEPENDS=y
# CONFIG_CHELSIO_T4VF is not set
# CONFIG_ENIC is not set
# CONFIG_IXGBE is not set
CONFIG_IXGB=m
CONFIG_S2IO=m
CONFIG_MYRI10GE=m
# CONFIG_NIU is not set
# CONFIG_MLX4_EN is not set
# CONFIG_MLX4_CORE is not set
# CONFIG_TEHUTI is not set
# CONFIG_BNX2X is not set
# CONFIG_QLCNIC is not set
# CONFIG_QLGE is not set
# CONFIG_SFC is not set
# CONFIG_BE2NET is not set
CONFIG_TR=y
CONFIG_IBMOL=m
CONFIG_3C359=m
# CONFIG_TMS380TR is not set
CONFIG_WLAN=y
# CONFIG_PCMCIA_RAYCS is not set
# CONFIG_AIRO is not set
# CONFIG_ATMEL is not set
# CONFIG_AIRO_CS is not set
# CONFIG_PCMCIA_WL3501 is not set
# CONFIG_PRISM54 is not set
# CONFIG_USB_ZD1201 is not set
# CONFIG_HOSTAP is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#

#
# USB Network Adapters
#
CONFIG_USB_CATC=m
CONFIG_USB_KAWETH=m
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=m
CONFIG_USB_USBNET=m
CONFIG_USB_NET_AX8817X=m
CONFIG_USB_NET_CDCETHER=m
# CONFIG_USB_NET_CDC_EEM is not set
CONFIG_USB_NET_DM9601=m
# CONFIG_USB_NET_SMSC75XX is not set
# CONFIG_USB_NET_SMSC95XX is not set
CONFIG_USB_NET_GL620A=m
CONFIG_USB_NET_NET1080=m
CONFIG_USB_NET_PLUSB=m
CONFIG_USB_NET_MCS7830=m
CONFIG_USB_NET_RNDIS_HOST=m
CONFIG_USB_NET_CDC_SUBSET=m
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
CONFIG_USB_KC2190=y
CONFIG_USB_NET_ZAURUS=m
# CONFIG_USB_HSO is not set
# CONFIG_USB_NET_INT51X1 is not set
# CONFIG_USB_IPHETH is not set
# CONFIG_USB_SIERRA_NET is not set
CONFIG_NET_PCMCIA=y
CONFIG_PCMCIA_3C589=m
CONFIG_PCMCIA_3C574=m
CONFIG_PCMCIA_FMVJ18X=m
CONFIG_PCMCIA_PCNET=m
CONFIG_PCMCIA_NMCLAN=m
CONFIG_PCMCIA_SMC91C92=m
CONFIG_PCMCIA_XIRC2PS=m
CONFIG_PCMCIA_AXNET=m
# CONFIG_PCMCIA_IBMTR is not set
# CONFIG_WAN is not set
CONFIG_ATM_DRIVERS=y
# CONFIG_ATM_DUMMY is not set
CONFIG_ATM_TCP=m
CONFIG_ATM_LANAI=m
CONFIG_ATM_ENI=m
# CONFIG_ATM_ENI_DEBUG is not set
# CONFIG_ATM_ENI_TUNE_BURST is not set
CONFIG_ATM_FIRESTREAM=m
# CONFIG_ATM_ZATM is not set
# CONFIG_ATM_NICSTAR is not set
CONFIG_ATM_IDT77252=m
# CONFIG_ATM_IDT77252_DEBUG is not set
# CONFIG_ATM_IDT77252_RCV_ALL is not set
CONFIG_ATM_IDT77252_USE_SUNI=y
CONFIG_ATM_AMBASSADOR=m
# CONFIG_ATM_AMBASSADOR_DEBUG is not set
CONFIG_ATM_HORIZON=m
# CONFIG_ATM_HORIZON_DEBUG is not set
# CONFIG_ATM_IA is not set
# CONFIG_ATM_FORE200E is not set
CONFIG_ATM_HE=m
# CONFIG_ATM_HE_USE_SUNI is not set
# CONFIG_ATM_SOLOS is not set

#
# CAIF transport drivers
#
CONFIG_FDDI=y
# CONFIG_DEFXX is not set
CONFIG_SKFP=m
# CONFIG_HIPPI is not set
CONFIG_PLIP=m
CONFIG_PPP=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
# CONFIG_PPP_BSDCOMP is not set
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=m
CONFIG_PPPOATM=m
CONFIG_SLIP=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLHC=m
CONFIG_SLIP_SMART=y
# CONFIG_SLIP_MODE_SLIP6 is not set
CONFIG_NET_FC=y
CONFIG_NETCONSOLE=y
# CONFIG_NETCONSOLE_DYNAMIC is not set
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
# CONFIG_VMXNET3 is not set
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_FF_MEMLESS=y
CONFIG_INPUT_POLLDEV=y
CONFIG_INPUT_SPARSEKMAP=m

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
# CONFIG_KEYBOARD_ADP5588 is not set
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_QT2160 is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_LM8323 is not set
# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_MCS is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
CONFIG_KEYBOARD_STOWAWAY=m
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_SENTELIC is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
CONFIG_MOUSE_SERIAL=m
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
CONFIG_MOUSE_VSXXXAA=m
# CONFIG_MOUSE_SYNAPTICS_I2C is not set
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=m
CONFIG_JOYSTICK_A3D=m
CONFIG_JOYSTICK_ADI=m
CONFIG_JOYSTICK_COBRA=m
CONFIG_JOYSTICK_GF2K=m
CONFIG_JOYSTICK_GRIP=m
CONFIG_JOYSTICK_GRIP_MP=m
CONFIG_JOYSTICK_GUILLEMOT=m
CONFIG_JOYSTICK_INTERACT=m
CONFIG_JOYSTICK_SIDEWINDER=m
CONFIG_JOYSTICK_TMDC=m
CONFIG_JOYSTICK_IFORCE=m
CONFIG_JOYSTICK_IFORCE_USB=y
CONFIG_JOYSTICK_IFORCE_232=y
CONFIG_JOYSTICK_WARRIOR=m
CONFIG_JOYSTICK_MAGELLAN=m
CONFIG_JOYSTICK_SPACEORB=m
CONFIG_JOYSTICK_SPACEBALL=m
CONFIG_JOYSTICK_STINGER=m
CONFIG_JOYSTICK_TWIDJOY=m
# CONFIG_JOYSTICK_ZHENHUA is not set
CONFIG_JOYSTICK_DB9=m
CONFIG_JOYSTICK_GAMECON=m
CONFIG_JOYSTICK_TURBOGRAFX=m
CONFIG_JOYSTICK_JOYDUMP=m
# CONFIG_JOYSTICK_XPAD is not set
# CONFIG_INPUT_TABLET is not set
CONFIG_INPUT_TOUCHSCREEN=y
# CONFIG_TOUCHSCREEN_AD7879 is not set
# CONFIG_TOUCHSCREEN_DYNAPRO is not set
# CONFIG_TOUCHSCREEN_HAMPSHIRE is not set
# CONFIG_TOUCHSCREEN_EETI is not set
# CONFIG_TOUCHSCREEN_FUJITSU is not set
CONFIG_TOUCHSCREEN_GUNZE=m
CONFIG_TOUCHSCREEN_ELO=m
# CONFIG_TOUCHSCREEN_WACOM_W8001 is not set
# CONFIG_TOUCHSCREEN_MCS5000 is not set
CONFIG_TOUCHSCREEN_MTOUCH=m
# CONFIG_TOUCHSCREEN_INEXIO is not set
CONFIG_TOUCHSCREEN_MK712=m
CONFIG_TOUCHSCREEN_PENMOUNT=m
# CONFIG_TOUCHSCREEN_QT602240 is not set
CONFIG_TOUCHSCREEN_TOUCHRIGHT=m
CONFIG_TOUCHSCREEN_TOUCHWIN=m
# CONFIG_TOUCHSCREEN_WM97XX is not set
# CONFIG_TOUCHSCREEN_USB_COMPOSITE is not set
# CONFIG_TOUCHSCREEN_TOUCHIT213 is not set
# CONFIG_TOUCHSCREEN_TSC2007 is not set
# CONFIG_TOUCHSCREEN_TPS6507X is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_AD714X is not set
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_APANEL is not set
CONFIG_INPUT_ATLAS_BTNS=m
# CONFIG_INPUT_ATI_REMOTE is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
CONFIG_INPUT_UINPUT=m
# CONFIG_INPUT_WINBOND_CIR is not set
# CONFIG_INPUT_PCF8574 is not set
# CONFIG_INPUT_ADXL34X is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
# CONFIG_SERIO_ALTERA_PS2 is not set
CONFIG_GAMEPORT=m
CONFIG_GAMEPORT_NS558=m
CONFIG_GAMEPORT_L4=m
CONFIG_GAMEPORT_EMU10K1=m
CONFIG_GAMEPORT_FM801=m

#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_DEVKMEM=y
CONFIG_SERIAL_NONSTANDARD=y
# CONFIG_COMPUTONE is not set
# CONFIG_ROCKETPORT is not set
CONFIG_CYCLADES=m
# CONFIG_CYZ_INTR is not set
# CONFIG_DIGIEPCA is not set
# CONFIG_MOXA_INTELLIO is not set
# CONFIG_MOXA_SMARTIO is not set
# CONFIG_ISI is not set
CONFIG_SYNCLINK=m
CONFIG_SYNCLINKMP=m
CONFIG_SYNCLINK_GT=m
CONFIG_N_HDLC=m
# CONFIG_N_GSM is not set
# CONFIG_RISCOM8 is not set
# CONFIG_SPECIALIX is not set
# CONFIG_STALDRV is not set
# CONFIG_NOZOMI is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_CS=m
CONFIG_SERIAL_8250_NR_UARTS=32
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_DETECT_IRQ=y
CONFIG_SERIAL_8250_RSA=y

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_MFD_HSU is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_JSM=m
# CONFIG_SERIAL_TIMBERDALE is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
# CONFIG_LEGACY_PTYS is not set
CONFIG_PRINTER=m
CONFIG_LP_CONSOLE=y
CONFIG_PPDEV=m
CONFIG_IPMI_HANDLER=m
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=y
# CONFIG_HW_RANDOM_TIMERIOMEM is not set
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_HW_RANDOM_VIA=y
CONFIG_NVRAM=y
CONFIG_R3964=m
# CONFIG_APPLICOM is not set

#
# PCMCIA character devices
#
# CONFIG_SYNCLINK_CS is not set
CONFIG_CARDMAN_4000=m
CONFIG_CARDMAN_4040=m
# CONFIG_IPWIRELESS is not set
CONFIG_MWAVE=m
# CONFIG_RAW_DRIVER is not set
# CONFIG_HPET is not set
CONFIG_HANGCHECK_TIMER=m
CONFIG_TCG_TPM=m
CONFIG_TCG_TIS=m
CONFIG_TCG_NSC=m
CONFIG_TCG_ATMEL=m
CONFIG_TCG_INFINEON=m
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
# CONFIG_RAMOOPS is not set
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
# CONFIG_I2C_MUX is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_SMBUS=m
CONFIG_I2C_ALGOBIT=m

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
# CONFIG_I2C_ISCH is not set
CONFIG_I2C_PIIX4=y
CONFIG_I2C_NFORCE2=y
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
CONFIG_I2C_SIS96X=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m

#
# ACPI drivers
#
# CONFIG_I2C_SCMI is not set

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PCA_PLATFORM is not set
# CONFIG_I2C_SIMTEC is not set
# CONFIG_I2C_XILINX is not set

#
# External I2C/SMBus adapter drivers
#
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_TINY_USB is not set

#
# Other I2C/SMBus bus drivers
#
CONFIG_I2C_STUB=m
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_SPI is not set

#
# PPS support
#
# CONFIG_PPS is not set
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
# CONFIG_GPIOLIB is not set
CONFIG_W1=m
CONFIG_W1_CON=y

#
# 1-wire Bus Masters
#
CONFIG_W1_MASTER_MATROX=m
CONFIG_W1_MASTER_DS2490=m
CONFIG_W1_MASTER_DS2482=m

#
# 1-wire Slaves
#
CONFIG_W1_SLAVE_THERM=m
CONFIG_W1_SLAVE_SMEM=m
# CONFIG_W1_SLAVE_DS2431 is not set
CONFIG_W1_SLAVE_DS2433=m
CONFIG_W1_SLAVE_DS2433_CRC=y
# CONFIG_W1_SLAVE_DS2760 is not set
# CONFIG_W1_SLAVE_BQ27000 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_TEST_POWER is not set
# CONFIG_BATTERY_DS2760 is not set
# CONFIG_BATTERY_DS2782 is not set
# CONFIG_BATTERY_BQ27x00 is not set
# CONFIG_BATTERY_MAX17040 is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=m
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Native drivers
#
CONFIG_SENSORS_ABITUGURU=m
# CONFIG_SENSORS_ABITUGURU3 is not set
# CONFIG_SENSORS_AD7414 is not set
# CONFIG_SENSORS_AD7418 is not set
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
CONFIG_SENSORS_ADM1029=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ADM9240=m
# CONFIG_SENSORS_ADT7411 is not set
# CONFIG_SENSORS_ADT7462 is not set
# CONFIG_SENSORS_ADT7470 is not set
# CONFIG_SENSORS_ADT7475 is not set
# CONFIG_SENSORS_ASC7621 is not set
CONFIG_SENSORS_K8TEMP=m
# CONFIG_SENSORS_K10TEMP is not set
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_ATXP1=m
CONFIG_SENSORS_DS1621=m
# CONFIG_SENSORS_I5K_AMB is not set
CONFIG_SENSORS_F71805F=m
# CONFIG_SENSORS_F71882FG is not set
# CONFIG_SENSORS_F75375S is not set
# CONFIG_SENSORS_FSCHMD is not set
# CONFIG_SENSORS_G760A is not set
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
# CONFIG_SENSORS_CORETEMP is not set
# CONFIG_SENSORS_PKGTEMP is not set
# CONFIG_SENSORS_IBMAEM is not set
# CONFIG_SENSORS_IBMPEX is not set
CONFIG_SENSORS_IT87=m
# CONFIG_SENSORS_JC42 is not set
CONFIG_SENSORS_LM63=m
# CONFIG_SENSORS_LM73 is not set
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
# CONFIG_SENSORS_LM93 is not set
# CONFIG_SENSORS_LTC4215 is not set
# CONFIG_SENSORS_LTC4245 is not set
# CONFIG_SENSORS_LM95241 is not set
CONFIG_SENSORS_MAX1619=m
# CONFIG_SENSORS_MAX6650 is not set
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_PC87427=m
CONFIG_SENSORS_PCF8591=m
CONFIG_SENSORS_SIS5595=m
# CONFIG_SENSORS_SMM665 is not set
# CONFIG_SENSORS_DME1737 is not set
# CONFIG_SENSORS_EMC1403 is not set
# CONFIG_SENSORS_EMC2103 is not set
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47M192=m
CONFIG_SENSORS_SMSC47B397=m
# CONFIG_SENSORS_ADS7828 is not set
# CONFIG_SENSORS_AMC6821 is not set
# CONFIG_SENSORS_THMC50 is not set
# CONFIG_SENSORS_TMP102 is not set
# CONFIG_SENSORS_TMP401 is not set
# CONFIG_SENSORS_TMP421 is not set
# CONFIG_SENSORS_VIA_CPUTEMP is not set
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_VT1211=m
CONFIG_SENSORS_VT8231=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
CONFIG_SENSORS_W83L785TS=m
# CONFIG_SENSORS_W83L786NG is not set
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
CONFIG_SENSORS_HDAPS=m
# CONFIG_SENSORS_LIS3_I2C is not set
# CONFIG_SENSORS_APPLESMC is not set

#
# ACPI drivers
#
# CONFIG_SENSORS_ATK0110 is not set
# CONFIG_SENSORS_LIS3LV02D is not set
CONFIG_THERMAL=y
# CONFIG_THERMAL_HWMON is not set
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set

#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
# CONFIG_F71808E_WDT is not set
# CONFIG_SC520_WDT is not set
# CONFIG_EUROTECH_WDT is not set
# CONFIG_IB700_WDT is not set
CONFIG_IBMASR=m
# CONFIG_WAFER_WDT is not set
CONFIG_I6300ESB_WDT=m
CONFIG_ITCO_WDT=m
CONFIG_ITCO_VENDOR_SUPPORT=y
# CONFIG_IT8712F_WDT is not set
# CONFIG_IT87_WDT is not set
# CONFIG_HP_WATCHDOG is not set
# CONFIG_SC1200_WDT is not set
CONFIG_PC87413_WDT=m
# CONFIG_60XX_WDT is not set
# CONFIG_SBC8360_WDT is not set
# CONFIG_CPU5_WDT is not set
# CONFIG_SMSC_SCH311X_WDT is not set
# CONFIG_SMSC37B787_WDT is not set
CONFIG_W83627HF_WDT=m
CONFIG_W83697HF_WDT=m
# CONFIG_W83697UG_WDT is not set
CONFIG_W83877F_WDT=m
CONFIG_W83977F_WDT=m
CONFIG_MACHZ_WDT=m
# CONFIG_SBC_EPX_C3_WATCHDOG is not set

#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=m
CONFIG_WDTPCI=m

#
# USB-based Watchdog Cards
#
CONFIG_USBPCWATCHDOG=m
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
CONFIG_SSB=m
CONFIG_SSB_SPROM=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
# CONFIG_SSB_B43_PCI_BRIDGE is not set
CONFIG_SSB_PCMCIAHOST_POSSIBLE=y
# CONFIG_SSB_PCMCIAHOST is not set
CONFIG_SSB_SDIOHOST_POSSIBLE=y
# CONFIG_SSB_SDIOHOST is not set
# CONFIG_SSB_DEBUG is not set
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y
CONFIG_MFD_SUPPORT=y
# CONFIG_MFD_CORE is not set
# CONFIG_MFD_88PM860X is not set
CONFIG_MFD_SM501=m
# CONFIG_HTC_PASIC3 is not set
# CONFIG_TPS6507X is not set
# CONFIG_TWL4030_CORE is not set
# CONFIG_MFD_STMPE is not set
# CONFIG_MFD_TC35892 is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_PMIC_DA903X is not set
# CONFIG_PMIC_ADP5520 is not set
# CONFIG_MFD_MAX8925 is not set
# CONFIG_MFD_MAX8998 is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_WM831X is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_MFD_WM8994 is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_ABX500_CORE is not set
# CONFIG_LPC_SCH is not set
# CONFIG_MFD_RDC321X is not set
# CONFIG_MFD_JANZ_CMODIO is not set
# CONFIG_REGULATOR is not set
# CONFIG_MEDIA_SUPPORT is not set

#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_AGP_SIS=y
CONFIG_AGP_VIA=y
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=16
# CONFIG_VGA_SWITCHEROO is not set
CONFIG_DRM=m
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_TTM=m
CONFIG_DRM_TDFX=m
CONFIG_DRM_R128=m
CONFIG_DRM_RADEON=m
CONFIG_DRM_I810=m
CONFIG_DRM_I830=m
CONFIG_DRM_I915=m
# CONFIG_DRM_I915_KMS is not set
CONFIG_DRM_MGA=m
# CONFIG_DRM_SIS is not set
CONFIG_DRM_VIA=m
CONFIG_DRM_SAVAGE=m
CONFIG_VGASTATE=m
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
CONFIG_FB_DDC=m
# CONFIG_FB_BOOT_VESA_SUPPORT is not set
CONFIG_FB_CFB_FILLRECT=m
CONFIG_FB_CFB_COPYAREA=m
CONFIG_FB_CFB_IMAGEBLIT=m
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
# CONFIG_FB_SYS_FILLRECT is not set
# CONFIG_FB_SYS_COPYAREA is not set
# CONFIG_FB_SYS_IMAGEBLIT is not set
# CONFIG_FB_FOREIGN_ENDIAN is not set
# CONFIG_FB_SYS_FOPS is not set
CONFIG_FB_SVGALIB=m
# CONFIG_FB_MACMODES is not set
CONFIG_FB_BACKLIGHT=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_UVESA is not set
# CONFIG_FB_VESA is not set
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
CONFIG_FB_NVIDIA=m
CONFIG_FB_NVIDIA_I2C=y
# CONFIG_FB_NVIDIA_DEBUG is not set
CONFIG_FB_NVIDIA_BACKLIGHT=y
CONFIG_FB_RIVA=m
# CONFIG_FB_RIVA_I2C is not set
# CONFIG_FB_RIVA_DEBUG is not set
CONFIG_FB_RIVA_BACKLIGHT=y
# CONFIG_FB_LE80578 is not set
CONFIG_FB_MATROX=m
CONFIG_FB_MATROX_MILLENIUM=y
CONFIG_FB_MATROX_MYSTIQUE=y
CONFIG_FB_MATROX_G=y
CONFIG_FB_MATROX_I2C=m
CONFIG_FB_MATROX_MAVEN=m
# CONFIG_FB_RADEON is not set
CONFIG_FB_ATY128=m
CONFIG_FB_ATY128_BACKLIGHT=y
CONFIG_FB_ATY=m
CONFIG_FB_ATY_CT=y
CONFIG_FB_ATY_GENERIC_LCD=y
CONFIG_FB_ATY_GX=y
CONFIG_FB_ATY_BACKLIGHT=y
CONFIG_FB_S3=m
CONFIG_FB_SAVAGE=m
CONFIG_FB_SAVAGE_I2C=y
CONFIG_FB_SAVAGE_ACCEL=y
# CONFIG_FB_SIS is not set
# CONFIG_FB_VIA is not set
CONFIG_FB_NEOMAGIC=m
CONFIG_FB_KYRO=m
CONFIG_FB_3DFX=m
CONFIG_FB_3DFX_ACCEL=y
CONFIG_FB_3DFX_I2C=y
CONFIG_FB_VOODOO1=m
# CONFIG_FB_VT8623 is not set
CONFIG_FB_TRIDENT=m
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_GEODE is not set
CONFIG_FB_SM501=m
# CONFIG_FB_VIRTUAL is not set
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_FB_BROADSHEET is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=m
# CONFIG_LCD_PLATFORM is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
CONFIG_BACKLIGHT_GENERIC=y
CONFIG_BACKLIGHT_PROGEAR=m
# CONFIG_BACKLIGHT_MBP_NVIDIA is not set
# CONFIG_BACKLIGHT_SAHARA is not set
# CONFIG_BACKLIGHT_ADP8860 is not set

#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=m
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=m
CONFIG_SOUND_OSS_CORE=y
CONFIG_SOUND_OSS_CORE_PRECLAIM=y
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_JACK=y
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_DYNAMIC_MINORS=y
# CONFIG_SND_SUPPORT_OLD_API is not set
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_RAWMIDI_SEQ=m
CONFIG_SND_OPL3_LIB_SEQ=m
# CONFIG_SND_OPL4_LIB_SEQ is not set
# CONFIG_SND_SBAWE_SEQ is not set
CONFIG_SND_EMU10K1_SEQ=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
CONFIG_SND_VX_LIB=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DRIVERS=y
CONFIG_SND_DUMMY=m
CONFIG_SND_VIRMIDI=m
CONFIG_SND_MTS64=m
# CONFIG_SND_SERIAL_U16550 is not set
CONFIG_SND_MPU401=m
CONFIG_SND_PORTMAN2X4=m
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=0
CONFIG_SND_SB_COMMON=m
CONFIG_SND_PCI=y
CONFIG_SND_AD1889=m
CONFIG_SND_ALS300=m
CONFIG_SND_ALS4000=m
CONFIG_SND_ALI5451=m
# CONFIG_SND_ASIHPI is not set
CONFIG_SND_ATIIXP=m
CONFIG_SND_ATIIXP_MODEM=m
CONFIG_SND_AU8810=m
CONFIG_SND_AU8820=m
CONFIG_SND_AU8830=m
# CONFIG_SND_AW2 is not set
CONFIG_SND_AZT3328=m
CONFIG_SND_BT87X=m
# CONFIG_SND_BT87X_OVERCLOCK is not set
CONFIG_SND_CA0106=m
CONFIG_SND_CMIPCI=m
# CONFIG_SND_OXYGEN is not set
CONFIG_SND_CS4281=m
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_CS5535AUDIO is not set
# CONFIG_SND_CTXFI is not set
CONFIG_SND_DARLA20=m
CONFIG_SND_GINA20=m
CONFIG_SND_LAYLA20=m
CONFIG_SND_DARLA24=m
CONFIG_SND_GINA24=m
CONFIG_SND_LAYLA24=m
CONFIG_SND_MONA=m
CONFIG_SND_MIA=m
CONFIG_SND_ECHO3G=m
CONFIG_SND_INDIGO=m
CONFIG_SND_INDIGOIO=m
CONFIG_SND_INDIGODJ=m
# CONFIG_SND_INDIGOIOX is not set
# CONFIG_SND_INDIGODJX is not set
CONFIG_SND_EMU10K1=m
CONFIG_SND_EMU10K1X=m
CONFIG_SND_ENS1370=m
CONFIG_SND_ENS1371=m
CONFIG_SND_ES1938=m
CONFIG_SND_ES1968=m
# CONFIG_SND_ES1968_INPUT is not set
CONFIG_SND_FM801=m
CONFIG_SND_HDA_INTEL=m
# CONFIG_SND_HDA_HWDEP is not set
# CONFIG_SND_HDA_INPUT_BEEP is not set
# CONFIG_SND_HDA_INPUT_JACK is not set
# CONFIG_SND_HDA_PATCH_LOADER is not set
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_ATIHDMI=y
CONFIG_SND_HDA_CODEC_NVHDMI=y
CONFIG_SND_HDA_CODEC_INTELHDMI=y
CONFIG_SND_HDA_ELD=y
CONFIG_SND_HDA_CODEC_CIRRUS=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CA0110=y
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
# CONFIG_SND_HDA_POWER_SAVE is not set
CONFIG_SND_HDSP=m
CONFIG_SND_HDSPM=m
# CONFIG_SND_HIFIER is not set
CONFIG_SND_ICE1712=m
CONFIG_SND_ICE1724=m
CONFIG_SND_INTEL8X0=m
CONFIG_SND_INTEL8X0M=m
CONFIG_SND_KORG1212=m
# CONFIG_SND_LX6464ES is not set
CONFIG_SND_MAESTRO3=m
# CONFIG_SND_MAESTRO3_INPUT is not set
CONFIG_SND_MIXART=m
CONFIG_SND_NM256=m
CONFIG_SND_PCXHR=m
CONFIG_SND_RIPTIDE=m
CONFIG_SND_RME32=m
CONFIG_SND_RME96=m
CONFIG_SND_RME9652=m
CONFIG_SND_SONICVIBES=m
CONFIG_SND_TRIDENT=m
CONFIG_SND_VIA82XX=m
CONFIG_SND_VIA82XX_MODEM=m
# CONFIG_SND_VIRTUOSO is not set
CONFIG_SND_VX222=m
CONFIG_SND_YMFPCI=m
CONFIG_SND_USB=y
CONFIG_SND_USB_AUDIO=m
# CONFIG_SND_USB_UA101 is not set
CONFIG_SND_USB_USX2Y=m
# CONFIG_SND_USB_CAIAQ is not set
# CONFIG_SND_USB_US122L is not set
CONFIG_SND_PCMCIA=y
# CONFIG_SND_VXPOCKET is not set
# CONFIG_SND_PDAUDIOCF is not set
CONFIG_SND_SOC=m
CONFIG_SND_SOC_I2C_AND_SPI=m
# CONFIG_SND_SOC_ALL_CODECS is not set
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=m
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
# CONFIG_HIDRAW is not set

#
# USB Input Devices
#
CONFIG_USB_HID=y
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y

#
# Special HID drivers
#
# CONFIG_HID_3M_PCT is not set
CONFIG_HID_A4TECH=y
# CONFIG_HID_ACRUX_FF is not set
CONFIG_HID_APPLE=y
CONFIG_HID_BELKIN=y
# CONFIG_HID_CANDO is not set
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
# CONFIG_HID_PRODIKEYS is not set
CONFIG_HID_CYPRESS=y
# CONFIG_HID_DRAGONRISE is not set
# CONFIG_HID_EGALAX is not set
# CONFIG_HID_ELECOM is not set
CONFIG_HID_EZKEY=y
CONFIG_HID_KYE=y
CONFIG_HID_GYRATION=y
# CONFIG_HID_TWINHAN is not set
CONFIG_HID_KENSINGTON=y
CONFIG_HID_LOGITECH=y
CONFIG_LOGITECH_FF=y
# CONFIG_LOGIRUMBLEPAD2_FF is not set
# CONFIG_LOGIG940_FF is not set
# CONFIG_HID_MAGICMOUSE is not set
CONFIG_HID_MICROSOFT=y
# CONFIG_HID_MOSART is not set
CONFIG_HID_MONTEREY=y
CONFIG_HID_NTRIG=y
# CONFIG_HID_ORTEK is not set
CONFIG_HID_PANTHERLORD=y
CONFIG_PANTHERLORD_FF=y
CONFIG_HID_PETALYNX=y
# CONFIG_HID_PICOLCD is not set
# CONFIG_HID_QUANTA is not set
# CONFIG_HID_ROCCAT is not set
# CONFIG_HID_ROCCAT_KONE is not set
CONFIG_HID_SAMSUNG=y
CONFIG_HID_SONY=y
# CONFIG_HID_STANTUM is not set
CONFIG_HID_SUNPLUS=y
# CONFIG_HID_GREENASIA is not set
# CONFIG_HID_SMARTJOYPLUS is not set
CONFIG_HID_TOPSEED=y
# CONFIG_HID_THRUSTMASTER is not set
# CONFIG_HID_WACOM is not set
# CONFIG_HID_ZEROPLUS is not set
# CONFIG_HID_ZYDACRON is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
CONFIG_USB_DEVICE_CLASS=y
# CONFIG_USB_DYNAMIC_MINORS is not set
CONFIG_USB_MON=y
# CONFIG_USB_WUSB is not set
# CONFIG_USB_WUSB_CBAF is not set

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
# CONFIG_USB_XHCI_HCD is not set
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
# CONFIG_USB_OXU210HP_HCD is not set
CONFIG_USB_ISP116X_HCD=m
# CONFIG_USB_ISP1760_HCD is not set
# CONFIG_USB_ISP1362_HCD is not set
CONFIG_USB_OHCI_HCD=y
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=y
CONFIG_USB_U132_HCD=m
CONFIG_USB_SL811_HCD=m
CONFIG_USB_SL811_CS=m
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_HWA_HCD is not set

#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
# CONFIG_USB_WDM is not set
# CONFIG_USB_TMC is not set

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#

#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
CONFIG_USB_STORAGE_DATAFAB=m
CONFIG_USB_STORAGE_FREECOM=m
CONFIG_USB_STORAGE_ISD200=m
CONFIG_USB_STORAGE_USBAT=m
CONFIG_USB_STORAGE_SDDR09=m
CONFIG_USB_STORAGE_SDDR55=m
CONFIG_USB_STORAGE_JUMPSHOT=m
CONFIG_USB_STORAGE_ALAUDA=m
# CONFIG_USB_STORAGE_ONETOUCH is not set
CONFIG_USB_STORAGE_KARMA=m
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
CONFIG_USB_LIBUSUAL=y

#
# USB Imaging devices
#
CONFIG_USB_MDC800=m
CONFIG_USB_MICROTEK=m

#
# USB port drivers
#
CONFIG_USB_USS720=m
CONFIG_USB_SERIAL=m
CONFIG_USB_EZUSB=y
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_AIRCABLE=m
CONFIG_USB_SERIAL_ARK3116=m
CONFIG_USB_SERIAL_BELKIN=m
# CONFIG_USB_SERIAL_CH341 is not set
CONFIG_USB_SERIAL_WHITEHEAT=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
# CONFIG_USB_SERIAL_CP210X is not set
CONFIG_USB_SERIAL_CYPRESS_M8=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_FUNSOFT=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
CONFIG_USB_SERIAL_GARMIN=m
CONFIG_USB_SERIAL_IPW=m
# CONFIG_USB_SERIAL_IUU is not set
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
CONFIG_USB_SERIAL_KEYSPAN_MPR=y
CONFIG_USB_SERIAL_KEYSPAN_USA28=y
CONFIG_USB_SERIAL_KEYSPAN_USA28X=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XA=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XB=y
CONFIG_USB_SERIAL_KEYSPAN_USA19=y
CONFIG_USB_SERIAL_KEYSPAN_USA18X=y
CONFIG_USB_SERIAL_KEYSPAN_USA19W=y
CONFIG_USB_SERIAL_KEYSPAN_USA19QW=y
CONFIG_USB_SERIAL_KEYSPAN_USA19QI=y
CONFIG_USB_SERIAL_KEYSPAN_USA49W=y
CONFIG_USB_SERIAL_KEYSPAN_USA49WLC=y
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_MOS7720=m
# CONFIG_USB_SERIAL_MOS7715_PARPORT is not set
CONFIG_USB_SERIAL_MOS7840=m
# CONFIG_USB_SERIAL_MOTOROLA is not set
CONFIG_USB_SERIAL_NAVMAN=m
CONFIG_USB_SERIAL_PL2303=m
# CONFIG_USB_SERIAL_OTI6858 is not set
# CONFIG_USB_SERIAL_QCAUX is not set
# CONFIG_USB_SERIAL_QUALCOMM is not set
# CONFIG_USB_SERIAL_SPCP8X5 is not set
CONFIG_USB_SERIAL_HP4X=m
CONFIG_USB_SERIAL_SAFE=m
CONFIG_USB_SERIAL_SAFE_PADDED=y
# CONFIG_USB_SERIAL_SIEMENS_MPI is not set
CONFIG_USB_SERIAL_SIERRAWIRELESS=m
# CONFIG_USB_SERIAL_SYMBOL is not set
CONFIG_USB_SERIAL_TI=m
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_WWAN=m
CONFIG_USB_SERIAL_OPTION=m
CONFIG_USB_SERIAL_OMNINET=m
# CONFIG_USB_SERIAL_OPTICON is not set
# CONFIG_USB_SERIAL_VIVOPAY_SERIAL is not set
# CONFIG_USB_SERIAL_ZIO is not set
# CONFIG_USB_SERIAL_SSU100 is not set
CONFIG_USB_SERIAL_DEBUG=m

#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=m
CONFIG_USB_EMI26=m
CONFIG_USB_ADUTUX=m
# CONFIG_USB_SEVSEG is not set
CONFIG_USB_RIO500=m
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=m
CONFIG_USB_LED=m
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
CONFIG_USB_IDMOUSE=m
CONFIG_USB_FTDI_ELAN=m
CONFIG_USB_APPLEDISPLAY=m
CONFIG_USB_SISUSBVGA=m
CONFIG_USB_SISUSBVGA_CON=y
CONFIG_USB_LD=m
CONFIG_USB_TRANCEVIBRATOR=m
CONFIG_USB_IOWARRIOR=m
CONFIG_USB_TEST=m
# CONFIG_USB_ISIGHTFW is not set
CONFIG_USB_ATM=m
CONFIG_USB_SPEEDTOUCH=m
CONFIG_USB_CXACRU=m
CONFIG_USB_UEAGLEATM=m
CONFIG_USB_XUSBATM=m
# CONFIG_USB_GADGET is not set

#
# OTG and related infrastructure
#
# CONFIG_NOP_USB_XCEIV is not set
# CONFIG_UWB is not set
CONFIG_MMC=m
# CONFIG_MMC_DEBUG is not set
# CONFIG_MMC_UNSAFE_RESUME is not set

#
# MMC/SD/SDIO Card Drivers
#
CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_BOUNCE=y
# CONFIG_SDIO_UART is not set
# CONFIG_MMC_TEST is not set

#
# MMC/SD/SDIO Host Controller Drivers
#
CONFIG_MMC_SDHCI=m
# CONFIG_MMC_SDHCI_PCI is not set
# CONFIG_MMC_SDHCI_PLTFM is not set
CONFIG_MMC_WBSD=m
CONFIG_MMC_TIFM_SD=m
# CONFIG_MMC_SDRICOH_CS is not set
# CONFIG_MMC_CB710 is not set
# CONFIG_MMC_VIA_SDMMC is not set
# CONFIG_MEMSTICK is not set
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y

#
# LED drivers
#
# CONFIG_LEDS_ALIX2 is not set
# CONFIG_LEDS_PCA9532 is not set
# CONFIG_LEDS_LP3944 is not set
# CONFIG_LEDS_CLEVO_MAIL is not set
# CONFIG_LEDS_PCA955X is not set
# CONFIG_LEDS_BD2802 is not set
CONFIG_LEDS_TRIGGERS=y

#
# LED Triggers
#
CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_HEARTBEAT=m
# CONFIG_LEDS_TRIGGER_BACKLIGHT is not set
# CONFIG_LEDS_TRIGGER_DEFAULT_ON is not set

#
# iptables trigger is under Netfilter config (LED target)
#
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
CONFIG_EDAC=y

#
# Reporting subsystems
#
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_DECODE_MCE=y
CONFIG_EDAC_MM_EDAC=m
# CONFIG_EDAC_AMD64 is not set
CONFIG_EDAC_E752X=m
# CONFIG_EDAC_I82975X is not set
# CONFIG_EDAC_I3000 is not set
# CONFIG_EDAC_I3200 is not set
# CONFIG_EDAC_X38 is not set
# CONFIG_EDAC_I5400 is not set
# CONFIG_EDAC_I7CORE is not set
# CONFIG_EDAC_I5000 is not set
# CONFIG_EDAC_I5100 is not set
CONFIG_RTC_LIB=m
CONFIG_RTC_CLASS=m

#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set

#
# I2C RTC drivers
#
CONFIG_RTC_DRV_DS1307=m
# CONFIG_RTC_DRV_DS1374 is not set
CONFIG_RTC_DRV_DS1672=m
# CONFIG_RTC_DRV_DS3232 is not set
# CONFIG_RTC_DRV_MAX6900 is not set
CONFIG_RTC_DRV_RS5C372=m
CONFIG_RTC_DRV_ISL1208=m
# CONFIG_RTC_DRV_ISL12022 is not set
CONFIG_RTC_DRV_X1205=m
CONFIG_RTC_DRV_PCF8563=m
# CONFIG_RTC_DRV_PCF8583 is not set
# CONFIG_RTC_DRV_M41T80 is not set
# CONFIG_RTC_DRV_BQ32K is not set
# CONFIG_RTC_DRV_S35390A is not set
# CONFIG_RTC_DRV_FM3130 is not set
# CONFIG_RTC_DRV_RX8581 is not set
# CONFIG_RTC_DRV_RX8025 is not set

#
# SPI RTC drivers
#

#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=m
# CONFIG_RTC_DRV_DS1286 is not set
# CONFIG_RTC_DRV_DS1511 is not set
CONFIG_RTC_DRV_DS1553=m
CONFIG_RTC_DRV_DS1742=m
# CONFIG_RTC_DRV_STK17TA8 is not set
# CONFIG_RTC_DRV_M48T86 is not set
# CONFIG_RTC_DRV_M48T35 is not set
# CONFIG_RTC_DRV_M48T59 is not set
# CONFIG_RTC_DRV_MSM6242 is not set
# CONFIG_RTC_DRV_BQ4802 is not set
# CONFIG_RTC_DRV_RP5C01 is not set
CONFIG_RTC_DRV_V3020=m

#
# on-CPU RTC drivers
#
# CONFIG_DMADEVICES is not set
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
# CONFIG_STAGING is not set
CONFIG_X86_PLATFORM_DEVICES=y
# CONFIG_ACER_WMI is not set
CONFIG_ASUS_LAPTOP=m
# CONFIG_DELL_LAPTOP is not set
# CONFIG_FUJITSU_LAPTOP is not set
CONFIG_MSI_LAPTOP=m
# CONFIG_PANASONIC_LAPTOP is not set
# CONFIG_COMPAL_LAPTOP is not set
CONFIG_SONY_LAPTOP=m
# CONFIG_SONYPI_COMPAT is not set
# CONFIG_IDEAPAD_ACPI is not set
# CONFIG_THINKPAD_ACPI is not set
# CONFIG_INTEL_MENLOW is not set
# CONFIG_EEEPC_LAPTOP is not set
# CONFIG_ACPI_WMI is not set
CONFIG_ACPI_ASUS=m
# CONFIG_TOPSTAR_LAPTOP is not set
CONFIG_ACPI_TOSHIBA=m
# CONFIG_TOSHIBA_BT_RFKILL is not set
# CONFIG_ACPI_CMPC is not set
# CONFIG_INTEL_IPS is not set

#
# Firmware Drivers
#
CONFIG_EDD=m
# CONFIG_EDD_OFF is not set
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_DELL_RBU=m
CONFIG_DCDBAS=m
CONFIG_DMIID=y
# CONFIG_ISCSI_IBFT_FIND is not set

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT2_FS_XIP=y
CONFIG_EXT3_FS=y
# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
# CONFIG_EXT4_FS is not set
CONFIG_FS_XIP=y
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=m
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=m
# CONFIG_REISERFS_CHECK is not set
CONFIG_REISERFS_PROC_INFO=y
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_JFS_FS=m
CONFIG_JFS_POSIX_ACL=y
CONFIG_JFS_SECURITY=y
# CONFIG_JFS_DEBUG is not set
# CONFIG_JFS_STATISTICS is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_XFS_FS=m
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
# CONFIG_XFS_RT is not set
# CONFIG_XFS_DEBUG is not set
CONFIG_GFS2_FS=m
# CONFIG_GFS2_FS_LOCKING_DLM is not set
CONFIG_OCFS2_FS=m
CONFIG_OCFS2_FS_O2CB=m
CONFIG_OCFS2_FS_USERSPACE_CLUSTER=m
CONFIG_OCFS2_FS_STATS=y
# CONFIG_OCFS2_DEBUG_MASKLOG is not set
# CONFIG_OCFS2_DEBUG_FS is not set
# CONFIG_BTRFS_FS is not set
# CONFIG_NILFS2_FS is not set
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_FANOTIFY is not set
CONFIG_QUOTA=y
# CONFIG_QUOTA_NETLINK_INTERFACE is not set
CONFIG_PRINT_QUOTA_WARNING=y
# CONFIG_QUOTA_DEBUG is not set
CONFIG_QUOTA_TREE=y
# CONFIG_QFMT_V1 is not set
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_QUOTACTL_COMPAT=y
CONFIG_AUTOFS_FS=m
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=m
# CONFIG_CUSE is not set
CONFIG_GENERIC_ACL=y

#
# Caches
#
# CONFIG_FSCACHE is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="ascii"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=m
CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ADFS_FS is not set
CONFIG_AFFS_FS=m
# CONFIG_ECRYPT_FS is not set
CONFIG_HFS_FS=m
CONFIG_HFSPLUS_FS=m
CONFIG_BEFS_FS=m
# CONFIG_BEFS_DEBUG is not set
CONFIG_BFS_FS=m
CONFIG_EFS_FS=m
# CONFIG_LOGFS is not set
CONFIG_CRAMFS=m
# CONFIG_SQUASHFS is not set
CONFIG_VXFS_FS=m
CONFIG_MINIX_FS=m
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
CONFIG_QNX4FS_FS=m
CONFIG_ROMFS_FS=m
CONFIG_ROMFS_BACKED_BY_BLOCK=y
CONFIG_ROMFS_ON_BLOCK=y
CONFIG_SYSV_FS=m
CONFIG_UFS_FS=m
# CONFIG_UFS_FS_WRITE is not set
# CONFIG_UFS_DEBUG is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_V4_1 is not set
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_RPCSEC_GSS_KRB5=m
CONFIG_RPCSEC_GSS_SPKM3=m
# CONFIG_SMB_FS is not set
# CONFIG_CEPH_FS is not set
CONFIG_CIFS=m
# CONFIG_CIFS_STATS is not set
CONFIG_CIFS_WEAK_PW_HASH=y
# CONFIG_CIFS_UPCALL is not set
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_DFS_UPCALL is not set
# CONFIG_CIFS_EXPERIMENTAL is not set
CONFIG_NCP_FS=m
CONFIG_NCPFS_PACKET_SIGNING=y
CONFIG_NCPFS_IOCTL_LOCKING=y
CONFIG_NCPFS_STRONG=y
CONFIG_NCPFS_NFS_NS=y
CONFIG_NCPFS_OS2_NS=y
CONFIG_NCPFS_SMALLDOS=y
CONFIG_NCPFS_NLS=y
CONFIG_NCPFS_EXTRAS=y
CONFIG_CODA_FS=m
# CONFIG_AFS_FS is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
# CONFIG_ATARI_PARTITION is not set
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
CONFIG_SGI_PARTITION=y
# CONFIG_ULTRIX_PARTITION is not set
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m
CONFIG_DLM_DEBUG=y

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_PRINTK_TIME is not set
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
# CONFIG_STRIP_ASM_SYMS is not set
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
# CONFIG_LOCKUP_DETECTOR is not set
# CONFIG_HARDLOCKUP_DETECTOR is not set
CONFIG_DETECT_HUNG_TASK=y
# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
CONFIG_RT_MUTEX_TESTER=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
# CONFIG_PROVE_LOCKING is not set
# CONFIG_SPARSE_RCU_POINTER is not set
CONFIG_LOCKDEP=y
CONFIG_LOCK_STAT=y
# CONFIG_DEBUG_LOCKDEP is not set
CONFIG_TRACE_IRQFLAGS=y
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# CONFIG_LKDTM is not set
# CONFIG_CPU_NOTIFIER_ERROR_INJECT is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
CONFIG_SYSCTL_SYSCALL_CHECK=y
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FTRACE_NMI_ENTER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_RING_BUFFER=y
CONFIG_FTRACE_NMI_ENTER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_IRQSOFF_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENT=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_FTRACE_MCOUNT_RECORD=y
CONFIG_FTRACE_SELFTEST=y
CONFIG_FTRACE_STARTUP_TEST=y
# CONFIG_EVENT_TRACE_TEST_SYSCALLS is not set
CONFIG_MMIOTRACE=y
# CONFIG_MMIOTRACE_TEST is not set
# CONFIG_RING_BUFFER_BENCHMARK is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
CONFIG_DYNAMIC_DEBUG=y
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_ATOMIC64_SELFTEST is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_HAVE_ARCH_KMEMCHECK=y
# CONFIG_STRICT_DEVMEM is not set
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_X86_PTDUMP is not set
CONFIG_DEBUG_RODATA=y
# CONFIG_DEBUG_RODATA_TEST is not set
# CONFIG_DEBUG_NX_TEST is not set
# CONFIG_IOMMU_DEBUG is not set
# CONFIG_IOMMU_STRESS is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
# CONFIG_X86_DECODER_SELFTEST is not set
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_DEBUG_BOOT_PARAMS is not set
# CONFIG_CPA_DEBUG is not set
CONFIG_OPTIMIZE_INLINING=y

#
# Security options
#
CONFIG_KEYS=y
CONFIG_KEYS_DEBUG_PROC_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
# CONFIG_SECURITY_NETWORK_XFRM is not set
# CONFIG_SECURITY_PATH is not set
CONFIG_LSM_MMAP_MIN_ADDR=65536
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
# CONFIG_SECURITY_SMACK is not set
# CONFIG_SECURITY_TOMOYO is not set
# CONFIG_SECURITY_APPARMOR is not set
# CONFIG_IMA is not set
CONFIG_DEFAULT_SECURITY_SELINUX=y
# CONFIG_DEFAULT_SECURITY_DAC is not set
CONFIG_DEFAULT_SECURITY="selinux"
CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
# CONFIG_ASYNC_RAID6_TEST is not set
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=m
CONFIG_CRYPTO_NULL=m
# CONFIG_CRYPTO_PCRYPT is not set
CONFIG_CRYPTO_WORKQUEUE=y
# CONFIG_CRYPTO_CRYPTD is not set
CONFIG_CRYPTO_AUTHENC=m
# CONFIG_CRYPTO_TEST is not set

#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_SEQIV is not set

#
# Block modes
#
CONFIG_CRYPTO_CBC=m
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_PCBC=m
# CONFIG_CRYPTO_XTS is not set

#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
# CONFIG_CRYPTO_VMAC is not set

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
# CONFIG_CRYPTO_CRC32C_INTEL is not set
# CONFIG_CRYPTO_GHASH is not set
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
# CONFIG_CRYPTO_RMD128 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
# CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL is not set

#
# Ciphers
#
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_AES_X86_64=m
# CONFIG_CRYPTO_AES_NI_INTEL is not set
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SALSA20_X86_64 is not set
# CONFIG_CRYPTO_SEED is not set
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m

#
# Compression
#
CONFIG_CRYPTO_DEFLATE=m
# CONFIG_CRYPTO_ZLIB is not set
# CONFIG_CRYPTO_LZO is not set

#
# Random Number Generation
#
# CONFIG_CRYPTO_ANSI_CPRNG is not set
CONFIG_CRYPTO_HW=y
# CONFIG_CRYPTO_DEV_PADLOCK is not set
# CONFIG_CRYPTO_DEV_HIFN_795X is not set
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_APIC_ARCHITECTURE=y
CONFIG_KVM_MMIO=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
# CONFIG_VHOST_NET is not set
# CONFIG_VIRTIO_PCI is not set
# CONFIG_VIRTIO_BALLOON is not set
CONFIG_BINARY_PRINTF=y

#
# Library routines
#
CONFIG_RAID6_PQ=m
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=m
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_LZO_DECOMPRESS=y
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_NLATTR=y

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Robert Richter]

On 11.09.10 08:45:37, Ingo Molnar wrote:
> 
> * Robert Richter <robert.richter@amd.com> wrote:
> 
> > On 11.09.10 05:41:57, Ingo Molnar wrote:
> > > Robert, can you reproduce the problem? It triggers rather easily on both 
> > > Peter's and my AMD systems.
> > 
> > I was testing 88a377e and so far it didn't trigger on the system I 
> > have available, will try other systems too. Can you send me your 
> > config too?
> 
> Sure, attached.

No, your config did not trigger it either. We will extend testing to
more systems, but I cannot start with it before tomorrow.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Robert Richter]

On 12.09.10 11:52:02, Robert Richter wrote:
> On 11.09.10 08:45:37, Ingo Molnar wrote:
> > 
> > * Robert Richter <robert.richter@amd.com> wrote:
> > 
> > > On 11.09.10 05:41:57, Ingo Molnar wrote:
> > > > Robert, can you reproduce the problem? It triggers rather easily on both 
> > > > Peter's and my AMD systems.
> > > 
> > > I was testing 88a377e and so far it didn't trigger on the system I 
> > > have available, will try other systems too. Can you send me your 
> > > config too?
> > 
> > Sure, attached.
> 
> No, your config did not trigger it either. We will extend testing to
> more systems, but I cannot start with it before tomorrow.

Ingo, Peter,

I finally found a system here, will start debugging

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Robert Richter]

On 13.09.10 16:37:13, Robert Richter wrote:
> Ingo, Peter,
> 
> I finally found a system here, will start debugging

I found the reason why we get the unknown nmi. For some reason
cpuc->active_mask in x86_pmu_handle_irq() is zero. Thus, no counters
are handled when we get an nmi. It seems there is somewhere a race
accessing the active_mask. So far I don't have a fix available.
Changing x86_pmu_stop() did not help:

static void x86_pmu_stop(struct perf_event *event, int flags)
{
        struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
        struct hw_perf_event *hwc = &event->hw;

        if (test_bit(hwc->idx, cpuc->active_mask)) {
                x86_pmu.disable(event);
                __clear_bit(hwc->idx, cpuc->active_mask);
                cpuc->events[hwc->idx] = NULL;
                WARN_ON_ONCE(hwc->state & PERF_HES_STOPPED);
                hwc->state |= PERF_HES_STOPPED;
        }
	...
}

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

[PATCH] perf, x86: catch spurious interrupts after disabling counters

[Robert Richter]

On 14.09.10 19:41:32, Robert Richter wrote:
> I found the reason why we get the unknown nmi. For some reason
> cpuc->active_mask in x86_pmu_handle_irq() is zero. Thus, no counters
> are handled when we get an nmi. It seems there is somewhere a race
> accessing the active_mask. So far I don't have a fix available.
> Changing x86_pmu_stop() did not help:

The patch below for tip/perf/urgent fixes this.

-Robert

>From 4206a086f5b37efc1b4d94f1d90b55802b299ca0 Mon Sep 17 00:00:00 2001
From: Robert Richter <robert.richter@amd.com>
Date: Wed, 15 Sep 2010 16:12:59 +0200
Subject: [PATCH] perf, x86: catch spurious interrupts after disabling counters

Some cpus still deliver spurious interrupts after disabling a counter.
This caused 'undelivered NMI' messages. This patch fixes this.

Signed-off-by: Robert Richter <robert.richter@amd.com>
---
 arch/x86/kernel/cpu/perf_event.c |   13 ++++++++++++-
 1 files changed, 12 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index 3efdf28..df7aabd 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -102,6 +102,7 @@ struct cpu_hw_events {
 	 */
 	struct perf_event	*events[X86_PMC_IDX_MAX]; /* in counter order */
 	unsigned long		active_mask[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
+	unsigned long		running[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
 	int			enabled;
 
 	int			n_events;
@@ -1010,6 +1011,7 @@ static int x86_pmu_start(struct perf_event *event)
 	x86_perf_event_set_period(event);
 	cpuc->events[idx] = event;
 	__set_bit(idx, cpuc->active_mask);
+	__set_bit(idx, cpuc->running);
 	x86_pmu.enable(event);
 	perf_event_update_userpage(event);
 
@@ -1141,8 +1143,17 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 	cpuc = &__get_cpu_var(cpu_hw_events);
 
 	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
-		if (!test_bit(idx, cpuc->active_mask))
+		if (!test_bit(idx, cpuc->active_mask)) {
+			if (__test_and_clear_bit(idx, cpuc->running))
+				/*
+				 * Though we deactivated the counter
+				 * some cpus might still deliver
+				 * spurious interrupts. Catching them
+				 * here.
+				 */
+				handled++;
 			continue;
+		}
 
 		event = cpuc->events[idx];
 		hwc = &event->hw;
-- 
1.7.2.2





-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Stephane Eranian]

On Wed, Sep 15, 2010 at 6:20 PM, Robert Richter <robert.richter@amd.com> wrote:
> On 14.09.10 19:41:32, Robert Richter wrote:
>> I found the reason why we get the unknown nmi. For some reason
>> cpuc->active_mask in x86_pmu_handle_irq() is zero. Thus, no counters
>> are handled when we get an nmi. It seems there is somewhere a race
>> accessing the active_mask. So far I don't have a fix available.
>> Changing x86_pmu_stop() did not help:
>
> The patch below for tip/perf/urgent fixes this.
>
> -Robert
>
> From 4206a086f5b37efc1b4d94f1d90b55802b299ca0 Mon Sep 17 00:00:00 2001
> From: Robert Richter <robert.richter@amd.com>
> Date: Wed, 15 Sep 2010 16:12:59 +0200
> Subject: [PATCH] perf, x86: catch spurious interrupts after disabling counters
>
> Some cpus still deliver spurious interrupts after disabling a counter.

Most likely the interrupt was in flight at the time you disabled it.
Does the counter value reflect this?
Were you also getting this if you were only measuring at the user level?

> This caused 'undelivered NMI' messages. This patch fixes this.
>
> Signed-off-by: Robert Richter <robert.richter@amd.com>
> ---
>  arch/x86/kernel/cpu/perf_event.c |   13 ++++++++++++-
>  1 files changed, 12 insertions(+), 1 deletions(-)
>
> diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
> index 3efdf28..df7aabd 100644
> --- a/arch/x86/kernel/cpu/perf_event.c
> +++ b/arch/x86/kernel/cpu/perf_event.c
> @@ -102,6 +102,7 @@ struct cpu_hw_events {
>         */
>        struct perf_event       *events[X86_PMC_IDX_MAX]; /* in counter order */
>        unsigned long           active_mask[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
> +       unsigned long           running[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
>        int                     enabled;
>
>        int                     n_events;
> @@ -1010,6 +1011,7 @@ static int x86_pmu_start(struct perf_event *event)
>        x86_perf_event_set_period(event);
>        cpuc->events[idx] = event;
>        __set_bit(idx, cpuc->active_mask);
> +       __set_bit(idx, cpuc->running);
>        x86_pmu.enable(event);
>        perf_event_update_userpage(event);
>
> @@ -1141,8 +1143,17 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
>        cpuc = &__get_cpu_var(cpu_hw_events);
>
>        for (idx = 0; idx < x86_pmu.num_counters; idx++) {
> -               if (!test_bit(idx, cpuc->active_mask))
> +               if (!test_bit(idx, cpuc->active_mask)) {
> +                       if (__test_and_clear_bit(idx, cpuc->running))
> +                               /*
> +                                * Though we deactivated the counter
> +                                * some cpus might still deliver
> +                                * spurious interrupts. Catching them
> +                                * here.
> +                                */
> +                               handled++;
>                        continue;
> +               }
>
>                event = cpuc->events[idx];
>                hwc = &event->hw;
> --
> 1.7.2.2
>
>
>
>
>
> --
> Advanced Micro Devices, Inc.
> Operating System Research Center
>
>

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Robert Richter]

On 15.09.10 12:36:27, Stephane Eranian wrote:
> On Wed, Sep 15, 2010 at 6:20 PM, Robert Richter <robert.richter@amd.com> wrote:
> > On 14.09.10 19:41:32, Robert Richter wrote:
> >> I found the reason why we get the unknown nmi. For some reason
> >> cpuc->active_mask in x86_pmu_handle_irq() is zero. Thus, no counters
> >> are handled when we get an nmi. It seems there is somewhere a race
> >> accessing the active_mask. So far I don't have a fix available.
> >> Changing x86_pmu_stop() did not help:
> >
> > The patch below for tip/perf/urgent fixes this.
> >
> > -Robert
> >
> > From 4206a086f5b37efc1b4d94f1d90b55802b299ca0 Mon Sep 17 00:00:00 2001
> > From: Robert Richter <robert.richter@amd.com>
> > Date: Wed, 15 Sep 2010 16:12:59 +0200
> > Subject: [PATCH] perf, x86: catch spurious interrupts after disabling counters
> >
> > Some cpus still deliver spurious interrupts after disabling a counter.
> 
> Most likely the interrupt was in flight at the time you disabled it.

I tried to clear the bit in the active_mask after disabling the
counter (writing to the msr), which did not solve it. Shouldn't the
counter be disabled immediatly? Maybe clearing the INT bit would have
been worked too, but I was not sure about side effects.

> Does the counter value reflect this?

Yes, the disabled bit was cleared after reading the evntsel msr and
the ctr value have had about 400 cycles (it could have been
overflowed, though we actually can't say since the counter was
disabled).

> Were you also getting this if you were only measuring at the user level?

I tried only

 perf record ./hackbench 10

which triggered it on my system.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Stephane Eranian]

On Wed, Sep 15, 2010 at 7:00 PM, Robert Richter <robert.richter@amd.com> wrote:
> On 15.09.10 12:36:27, Stephane Eranian wrote:
>> On Wed, Sep 15, 2010 at 6:20 PM, Robert Richter <robert.richter@amd.com> wrote:
>> > On 14.09.10 19:41:32, Robert Richter wrote:
>> >> I found the reason why we get the unknown nmi. For some reason
>> >> cpuc->active_mask in x86_pmu_handle_irq() is zero. Thus, no counters
>> >> are handled when we get an nmi. It seems there is somewhere a race
>> >> accessing the active_mask. So far I don't have a fix available.
>> >> Changing x86_pmu_stop() did not help:
>> >
>> > The patch below for tip/perf/urgent fixes this.
>> >
>> > -Robert
>> >
>> > From 4206a086f5b37efc1b4d94f1d90b55802b299ca0 Mon Sep 17 00:00:00 2001
>> > From: Robert Richter <robert.richter@amd.com>
>> > Date: Wed, 15 Sep 2010 16:12:59 +0200
>> > Subject: [PATCH] perf, x86: catch spurious interrupts after disabling counters
>> >
>> > Some cpus still deliver spurious interrupts after disabling a counter.
>>
>> Most likely the interrupt was in flight at the time you disabled it.
>
> I tried to clear the bit in the active_mask after disabling the
> counter (writing to the msr), which did not solve it. Shouldn't the
> counter be disabled immediatly? Maybe clearing the INT bit would have
> been worked too, but I was not sure about side effects.
>
0 instr1
1 instr2
2 instr3
3 wrmsrl(eventsel0, 0);

There is skid between the instruction you overflow the counter and
where the interrupt
is posted.  If you overflow on instr1, suppose you post the interrupt
on instr3 which
is immediately followed by disable. There may a chance you get the
interrupt even
though the counter was disabled. I also don't know when the INT bit is
looked at.

It may be worthwhile trying with:

static inline void x86_pmu_disable_event(struct perf_event *event)
{
        struct hw_perf_event *hwc = &event->hw;
        (void)checking_wrmsrl(hwc->config_base + hwc->idx, 0);
}

to see if it makes a difference.


>> Does the counter value reflect this?
>
> Yes, the disabled bit was cleared after reading the evntsel msr and
> the ctr value have had about 400 cycles (it could have been
> overflowed, though we actually can't say since the counter was
> disabled).
>
>> Were you also getting this if you were only measuring at the user level?
>
> I tried only
>
>  perf record ./hackbench 10
>
> which triggered it on my system.
>
I suspect that if you do:

perf record -e cycles:u ./hackbench 10

It does not happen.

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Robert Richter]

On 15.09.10 13:32:49, Stephane Eranian wrote:
> > I tried to clear the bit in the active_mask after disabling the
> > counter (writing to the msr), which did not solve it. Shouldn't the
> > counter be disabled immediatly? Maybe clearing the INT bit would have
> > been worked too, but I was not sure about side effects.
> >
> 0 instr1
> 1 instr2
> 2 instr3
> 3 wrmsrl(eventsel0, 0);
> 
> There is skid between the instruction you overflow the counter and
> where the interrupt
> is posted.  If you overflow on instr1, suppose you post the interrupt
> on instr3 which
> is immediately followed by disable. There may a chance you get the
> interrupt even
> though the counter was disabled. I also don't know when the INT bit is
> looked at.

Yes, this could be possible. So, we should assume interrupts may be
delivered after a counter is disabled, which the patch addresses.

> 
> It may be worthwhile trying with:
> 
> static inline void x86_pmu_disable_event(struct perf_event *event)
> {
>         struct hw_perf_event *hwc = &event->hw;
>         (void)checking_wrmsrl(hwc->config_base + hwc->idx, 0);
> }
> 
> to see if it makes a difference.
> 
> 
> >> Does the counter value reflect this?
> >
> > Yes, the disabled bit was cleared after reading the evntsel msr and
> > the ctr value have had about 400 cycles (it could have been
> > overflowed, though we actually can't say since the counter was
> > disabled).
> >
> >> Were you also getting this if you were only measuring at the user level?
> >
> > I tried only
> >
> >  perf record ./hackbench 10
> >
> > which triggered it on my system.
> >
> I suspect that if you do:
> 
> perf record -e cycles:u ./hackbench 10
> 
> It does not happen.

Do you know at which period the counters running for the following?

 perf record ./hackbench 10
 perf record -e cycles -e instructions -e cache-references \
     -e cache-misses -e branch-misses -a -- <cmd>

I couldn't find something about this in the man page.

I will do some further investigations here, esp. with:

* compile order,
* checking_wrmsrl(),
* -e cycles:u

But I can not start with it before next week.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Cyrill Gorcunov]

On Wed, Sep 15, 2010 at 08:44:24PM +0200, Robert Richter wrote:
...
> 
> Do you know at which period the counters running for the following?
> 
>  perf record ./hackbench 10
>  perf record -e cycles -e instructions -e cache-references \
>      -e cache-misses -e branch-misses -a -- <cmd>
> 
> I couldn't find something about this in the man page.
> 

It's on top of tools/perf/builtin-record.c so by default they
should be sampled with frequency 1KHz. Frederic or Peter knows
for sure ;)

	-- Cyrill

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Stephane Eranian]

When you do perf record foo, it's equivalent to
perf record -e cycles:uk -F 1000 foo

I think there is a problem with perf record in case you provide
multiple events. It is going to sample on all of them at the same
frequency. You may not always want that, but I don't think there
is a way to change that.


On Wed, Sep 15, 2010 at 9:34 PM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> On Wed, Sep 15, 2010 at 08:44:24PM +0200, Robert Richter wrote:
> ...
>>
>> Do you know at which period the counters running for the following?
>>
>>  perf record ./hackbench 10
>>  perf record -e cycles -e instructions -e cache-references \
>>      -e cache-misses -e branch-misses -a -- <cmd>
>>
>> I couldn't find something about this in the man page.
>>
>
> It's on top of tools/perf/builtin-record.c so by default they
> should be sampled with frequency 1KHz. Frederic or Peter knows
> for sure ;)
>
>        -- Cyrill
>

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Cyrill Gorcunov]

On Wed, Sep 15, 2010 at 10:21:10PM +0200, Stephane Eranian wrote:
> When you do perf record foo, it's equivalent to
> perf record -e cycles:uk -F 1000 foo
>

yup, thanks
 
> I think there is a problem with perf record in case you provide
> multiple events. It is going to sample on all of them at the same
> frequency. You may not always want that, but I don't think there
> is a way to change that.
> 

indeed, without perf-tool code changes but i think you still may run
multiple instances of perf-tool :)

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Robert Richter]

> On Wed, Sep 15, 2010 at 10:21:10PM +0200, Stephane Eranian wrote:
> > When you do perf record foo, it's equivalent to
> > perf record -e cycles:uk -F 1000 foo

Yes, thanks.

I am asking because I have observed up to 4 back-to-back nmis from the
same counter when enabling an event. The period is not yet adjusted.
We should avoid those short sampling periods in the beginning and
better start with too long periods. Didn't look at the implementation
so far. I know this is not easy to handle because this very much
depends on the event we measure.

Maybe we start the counter with a delay and then calculate period =
duration - delay, later decreasing the delay until the frequency is
adjusted but keeping the total sampling rate more or less constant.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Frederic Weisbecker]

On Thu, Sep 16, 2010 at 12:27:12AM +0200, Robert Richter wrote:
> > On Wed, Sep 15, 2010 at 10:21:10PM +0200, Stephane Eranian wrote:
> > > When you do perf record foo, it's equivalent to
> > > perf record -e cycles:uk -F 1000 foo
> 
> Yes, thanks.
> 
> I am asking because I have observed up to 4 back-to-back nmis from the
> same counter when enabling an event. The period is not yet adjusted.
> We should avoid those short sampling periods in the beginning and
> better start with too long periods. Didn't look at the implementation
> so far. I know this is not easy to handle because this very much
> depends on the event we measure.
> 
> Maybe we start the counter with a delay and then calculate period =
> duration - delay, later decreasing the delay until the frequency is
> adjusted but keeping the total sampling rate more or less constant.


Yep, IIRC, the first arming is set with a 1 sample period. So that a first
calculation can be made on top of the first triggering event time to get
a better sample period. That avoids the need to start with frequency
nature assumptions for each event.

I doubt that can be worked around easily.

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Cyrill Gorcunov]

On Wed, Sep 15, 2010 at 06:20:34PM +0200, Robert Richter wrote:
> On 14.09.10 19:41:32, Robert Richter wrote:
> > I found the reason why we get the unknown nmi. For some reason
> > cpuc->active_mask in x86_pmu_handle_irq() is zero. Thus, no counters
> > are handled when we get an nmi. It seems there is somewhere a race
> > accessing the active_mask. So far I don't have a fix available.
> > Changing x86_pmu_stop() did not help:
> 
> The patch below for tip/perf/urgent fixes this.
> 
> -Robert
> 
> From 4206a086f5b37efc1b4d94f1d90b55802b299ca0 Mon Sep 17 00:00:00 2001
> From: Robert Richter <robert.richter@amd.com>
> Date: Wed, 15 Sep 2010 16:12:59 +0200
> Subject: [PATCH] perf, x86: catch spurious interrupts after disabling counters
> 
> Some cpus still deliver spurious interrupts after disabling a counter.
> This caused 'undelivered NMI' messages. This patch fixes this.
> 
> Signed-off-by: Robert Richter <robert.richter@amd.com>
> ---
...

Hi Robert, thanks a lot for tracking this issue! I might be missing
something but why don't you clean this ->running mask bits on pmu-stop?
What if counter gets disabled/freed or whatever before issue any nmis?
Another question I have still -- is this an hardware issue in general?

	-- Cyrill

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Stephane Eranian]

On Wed, Sep 15, 2010 at 6:46 PM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> On Wed, Sep 15, 2010 at 06:20:34PM +0200, Robert Richter wrote:
>> On 14.09.10 19:41:32, Robert Richter wrote:
>> > I found the reason why we get the unknown nmi. For some reason
>> > cpuc->active_mask in x86_pmu_handle_irq() is zero. Thus, no counters
>> > are handled when we get an nmi. It seems there is somewhere a race
>> > accessing the active_mask. So far I don't have a fix available.
>> > Changing x86_pmu_stop() did not help:
>>
>> The patch below for tip/perf/urgent fixes this.
>>
>> -Robert
>>
>> From 4206a086f5b37efc1b4d94f1d90b55802b299ca0 Mon Sep 17 00:00:00 2001
>> From: Robert Richter <robert.richter@amd.com>
>> Date: Wed, 15 Sep 2010 16:12:59 +0200
>> Subject: [PATCH] perf, x86: catch spurious interrupts after disabling counters
>>
>> Some cpus still deliver spurious interrupts after disabling a counter.
>> This caused 'undelivered NMI' messages. This patch fixes this.
>>
>> Signed-off-by: Robert Richter <robert.richter@amd.com>
>> ---
> ...
>
> Hi Robert, thanks a lot for tracking this issue! I might be missing
> something but why don't you clean this ->running mask bits on pmu-stop?
> What if counter gets disabled/freed or whatever before issue any nmis?
> Another question I have still -- is this an hardware issue in general?
>
what's for sure, is that you can have an interrupt in flight by the time
you disable.

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Cyrill Gorcunov]

On Wed, Sep 15, 2010 at 06:47:51PM +0200, Stephane Eranian wrote:
> On Wed, Sep 15, 2010 at 6:46 PM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> > On Wed, Sep 15, 2010 at 06:20:34PM +0200, Robert Richter wrote:
> >> On 14.09.10 19:41:32, Robert Richter wrote:
> >> > I found the reason why we get the unknown nmi. For some reason
> >> > cpuc->active_mask in x86_pmu_handle_irq() is zero. Thus, no counters
> >> > are handled when we get an nmi. It seems there is somewhere a race
> >> > accessing the active_mask. So far I don't have a fix available.
> >> > Changing x86_pmu_stop() did not help:
> >>
> >> The patch below for tip/perf/urgent fixes this.
> >>
> >> -Robert
> >>
> >> From 4206a086f5b37efc1b4d94f1d90b55802b299ca0 Mon Sep 17 00:00:00 2001
> >> From: Robert Richter <robert.richter@amd.com>
> >> Date: Wed, 15 Sep 2010 16:12:59 +0200
> >> Subject: [PATCH] perf, x86: catch spurious interrupts after disabling counters
> >>
> >> Some cpus still deliver spurious interrupts after disabling a counter.
> >> This caused 'undelivered NMI' messages. This patch fixes this.
> >>
> >> Signed-off-by: Robert Richter <robert.richter@amd.com>
> >> ---
> > ...
> >
> > Hi Robert, thanks a lot for tracking this issue! I might be missing
> > something but why don't you clean this ->running mask bits on pmu-stop?
> > What if counter gets disabled/freed or whatever before issue any nmis?
> > Another question I have still -- is this an hardware issue in general?
> >
> what's for sure, is that you can have an interrupt in flight by the time
> you disable.
> 

I fear you can x86_pmu_stop()

	if (__test_and_clear_bit(hwc->idx, cpuc->active_mask)) {

--->		active_mask will be cleared here for sure
--->		but counter still ticks, say nmi happens active_mask
--->		is cleared, but NMI can still happen and gets buffered
--->		before you disable counter in real

		x86_pmu.disable(event);
		cpuc->events[hwc->idx] = NULL;
		WARN_ON_ONCE(hwc->state & PERF_HES_STOPPED);
		hwc->state |= PERF_HES_STOPPED;
	}

No?

	-- Cyrill

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Robert Richter]

On 15.09.10 13:02:22, Cyrill Gorcunov wrote:
> > what's for sure, is that you can have an interrupt in flight by the time
> > you disable.
> > 
> 
> I fear you can x86_pmu_stop()
> 
> 	if (__test_and_clear_bit(hwc->idx, cpuc->active_mask)) {
> 
> --->		active_mask will be cleared here for sure
> --->		but counter still ticks, say nmi happens active_mask
> --->		is cleared, but NMI can still happen and gets buffered
> --->		before you disable counter in real
> 
> 		x86_pmu.disable(event);
> 		cpuc->events[hwc->idx] = NULL;
> 		WARN_ON_ONCE(hwc->state & PERF_HES_STOPPED);
> 		hwc->state |= PERF_HES_STOPPED;
> 	}
> 
> No?

I tried reordering this too, but it didn't fix it.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Cyrill Gorcunov]

On Wed, Sep 15, 2010 at 07:28:05PM +0200, Robert Richter wrote:
> On 15.09.10 13:02:22, Cyrill Gorcunov wrote:
> > > what's for sure, is that you can have an interrupt in flight by the time
> > > you disable.
> > > 
> > 
> > I fear you can x86_pmu_stop()
> > 
> > 	if (__test_and_clear_bit(hwc->idx, cpuc->active_mask)) {
> > 
> > --->		active_mask will be cleared here for sure
> > --->		but counter still ticks, say nmi happens active_mask
> > --->		is cleared, but NMI can still happen and gets buffered
> > --->		before you disable counter in real
> > 
> > 		x86_pmu.disable(event);
> > 		cpuc->events[hwc->idx] = NULL;
> > 		WARN_ON_ONCE(hwc->state & PERF_HES_STOPPED);
> > 		hwc->state |= PERF_HES_STOPPED;
> > 	}
> > 
> > No?
> 
> I tried reordering this too, but it didn't fix it.
> 
> -Robert
>

Yeah, already noted from your previous email. Perhaps we might
do a bit simplier approach then -- in nmi handler were we mark
"next nmi" we could take into account not "one next" nmi but
sum of handled counters minus one being just handled (of course
cleaning this counter if new "non spurious" nmi came in), can't
say I like this approach but just a thought.

	-- Cyrill

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Robert Richter]

On 15.09.10 13:40:12, Cyrill Gorcunov wrote:
> Yeah, already noted from your previous email. Perhaps we might
> do a bit simplier approach then -- in nmi handler were we mark
> "next nmi" we could take into account not "one next" nmi but
> sum of handled counters minus one being just handled (of course
> cleaning this counter if new "non spurious" nmi came in), can't
> say I like this approach but just a thought.

If we disable a counter, it might still trigger an interrupt which we
cannot detect. Thus, if a running counter is deactivated, we must
count it as handled in the nmi handler.

Working with a sum is not possible, because a disabled counter may or
*may not* trigger an interrupt. We cannot predict the number of
counters that will be handled.

Dealing with the "next nmi" is also not handy here. Spurious nmis are
caused then stopping a counter. Since this is done outside the nmi
handler, we would then start touching the "next nmi" also outside the
handler. This might be more complex because we then have to deal with
locking or atomic access. We shouldn't do that.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Cyrill Gorcunov]

On Thu, Sep 16, 2010 at 12:10:41AM +0200, Robert Richter wrote:
> On 15.09.10 13:40:12, Cyrill Gorcunov wrote:
> > Yeah, already noted from your previous email. Perhaps we might
> > do a bit simplier approach then -- in nmi handler were we mark
> > "next nmi" we could take into account not "one next" nmi but
> > sum of handled counters minus one being just handled (of course
> > cleaning this counter if new "non spurious" nmi came in), can't
> > say I like this approach but just a thought.
> 
> If we disable a counter, it might still trigger an interrupt which we
> cannot detect. Thus, if a running counter is deactivated, we must
> count it as handled in the nmi handler.
> 
> Working with a sum is not possible, because a disabled counter may or
> *may not* trigger an interrupt. We cannot predict the number of
> counters that will be handled.
> 
> Dealing with the "next nmi" is also not handy here. Spurious nmis are
> caused then stopping a counter. Since this is done outside the nmi
> handler, we would then start touching the "next nmi" also outside the
> handler. This might be more complex because we then have to deal with
> locking or atomic access. We shouldn't do that.
> 
> -Robert
>

OK, I see what you mean Robert. Btw, when you reorder cpu_active_mask access
and wrmsr did you try also additional read after write of msr? ie like

	wrmsr
	barrier() // just to be sure gcc would not reorder it
	rdmsr
	clear cpu_active_mask

wonders if it did the trick

	-- Cyrill

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Peter Zijlstra]

On Wed, 2010-09-15 at 18:20 +0200, Robert Richter wrote:
> Some cpus still deliver spurious interrupts after disabling a counter.
> This caused 'undelivered NMI' messages. This patch fixes this.
> 
I tried the below and that also seems to work.. So yeah, looks like
we're getting late NMIs.

---
 arch/x86/kernel/cpu/perf_event.c |   21 ++++++++++++++++++++-
 1 files changed, 20 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index 0fb1705..9a261ac 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1145,6 +1145,22 @@ static void x86_pmu_del(struct perf_event *event, int flags)
 	perf_event_update_userpage(event);
 }
 
+static int fixup_overflow(int idx)
+{
+	u64 val;
+
+	rdmsrl(x86_pmu.perfctr + idx, val);
+	if (!(val & (1ULL << (x86_pmu.cntval_bits - 1)))) {
+		val = (u64)(-x86_pmu.max_period);
+		val &= x86_pmu.cntval_mask;
+		wrmsrl(x86_pmu.perfctr + idx, val);
+
+		return 1;
+	}
+
+	return 0;
+}
+
 static int x86_pmu_handle_irq(struct pt_regs *regs)
 {
 	struct perf_sample_data data;
@@ -1159,8 +1175,11 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 	cpuc = &__get_cpu_var(cpu_hw_events);
 
 	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
-		if (!test_bit(idx, cpuc->active_mask))
+		if (!test_bit(idx, cpuc->active_mask)) {
+	       		if (fixup_overflow(idx))
+				handled++;
 			continue;
+		}
 
 		event = cpuc->events[idx];
 		hwc = &event->hw;

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Robert Richter]

On 16.09.10 13:34:40, Peter Zijlstra wrote:
> On Wed, 2010-09-15 at 18:20 +0200, Robert Richter wrote:
> > Some cpus still deliver spurious interrupts after disabling a counter.
> > This caused 'undelivered NMI' messages. This patch fixes this.
> > 
> I tried the below and that also seems to work.. So yeah, looks like
> we're getting late NMIs.

I would rather prefer the fix I sent. This patch does a rdmsrl() with
each nmi on every inactive counter. It also changes the counter value
of all inactive counters, thus restarting a counter by only setting
the enable bit may start with an unexpected counter value (didn't look
at current implementation if this could be a problem).

It is also not possible to detect with hardware, which counter fired
the interrupt. We cannot assume a counter overflowed by just reading
the upper bit of the counter value. We must track this in software.

-Robert

> 
> ---
>  arch/x86/kernel/cpu/perf_event.c |   21 ++++++++++++++++++++-
>  1 files changed, 20 insertions(+), 1 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
> index 0fb1705..9a261ac 100644
> --- a/arch/x86/kernel/cpu/perf_event.c
> +++ b/arch/x86/kernel/cpu/perf_event.c
> @@ -1145,6 +1145,22 @@ static void x86_pmu_del(struct perf_event *event, int flags)
>  	perf_event_update_userpage(event);
>  }
>  
> +static int fixup_overflow(int idx)
> +{
> +	u64 val;
> +
> +	rdmsrl(x86_pmu.perfctr + idx, val);
> +	if (!(val & (1ULL << (x86_pmu.cntval_bits - 1)))) {
> +		val = (u64)(-x86_pmu.max_period);
> +		val &= x86_pmu.cntval_mask;
> +		wrmsrl(x86_pmu.perfctr + idx, val);
> +
> +		return 1;
> +	}
> +
> +	return 0;
> +}
> +
>  static int x86_pmu_handle_irq(struct pt_regs *regs)
>  {
>  	struct perf_sample_data data;
> @@ -1159,8 +1175,11 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
>  	cpuc = &__get_cpu_var(cpu_hw_events);
>  
>  	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
> -		if (!test_bit(idx, cpuc->active_mask))
> +		if (!test_bit(idx, cpuc->active_mask)) {
> +	       		if (fixup_overflow(idx))
> +				handled++;
>  			continue;
> +		}
>  
>  		event = cpuc->events[idx];
>  		hwc = &event->hw;
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Peter Zijlstra]

On Fri, 2010-09-17 at 10:51 +0200, Robert Richter wrote:
> On 16.09.10 13:34:40, Peter Zijlstra wrote:
> > On Wed, 2010-09-15 at 18:20 +0200, Robert Richter wrote:
> > > Some cpus still deliver spurious interrupts after disabling a counter.
> > > This caused 'undelivered NMI' messages. This patch fixes this.
> > > 
> > I tried the below and that also seems to work.. So yeah, looks like
> > we're getting late NMIs.
> 
> I would rather prefer the fix I sent. This patch does a rdmsrl() with
> each nmi on every inactive counter.

Sure, I was just playing around trying to see if that was indeed the
problem.

>  It also changes the counter value
> of all inactive counters, thus restarting a counter by only setting
> the enable bit may start with an unexpected counter value (didn't look
> at current implementation if this could be a problem).

It actually would, pmu->stop()/->start() won't save/restore the counter
value unless you add PERF_EF_UPDATE/PERF_EF_RELOAD.

> It is also not possible to detect with hardware, which counter fired
> the interrupt. We cannot assume a counter overflowed by just reading
> the upper bit of the counter value. We must track this in software. 

Well, exactly that seemed sufficient to not get spurious NMIs.

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Stephane Eranian]

Robert.

Does it mean that with this patch, we don't need Don's back-to-back NMI patch
anymore?


On Wed, Sep 15, 2010 at 6:20 PM, Robert Richter <robert.richter@amd.com> wrote:
> On 14.09.10 19:41:32, Robert Richter wrote:
>> I found the reason why we get the unknown nmi. For some reason
>> cpuc->active_mask in x86_pmu_handle_irq() is zero. Thus, no counters
>> are handled when we get an nmi. It seems there is somewhere a race
>> accessing the active_mask. So far I don't have a fix available.
>> Changing x86_pmu_stop() did not help:
>
> The patch below for tip/perf/urgent fixes this.
>
> -Robert
>
> From 4206a086f5b37efc1b4d94f1d90b55802b299ca0 Mon Sep 17 00:00:00 2001
> From: Robert Richter <robert.richter@amd.com>
> Date: Wed, 15 Sep 2010 16:12:59 +0200
> Subject: [PATCH] perf, x86: catch spurious interrupts after disabling counters
>
> Some cpus still deliver spurious interrupts after disabling a counter.
> This caused 'undelivered NMI' messages. This patch fixes this.
>
> Signed-off-by: Robert Richter <robert.richter@amd.com>
> ---
>  arch/x86/kernel/cpu/perf_event.c |   13 ++++++++++++-
>  1 files changed, 12 insertions(+), 1 deletions(-)
>
> diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
> index 3efdf28..df7aabd 100644
> --- a/arch/x86/kernel/cpu/perf_event.c
> +++ b/arch/x86/kernel/cpu/perf_event.c
> @@ -102,6 +102,7 @@ struct cpu_hw_events {
>         */
>        struct perf_event       *events[X86_PMC_IDX_MAX]; /* in counter order */
>        unsigned long           active_mask[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
> +       unsigned long           running[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
>        int                     enabled;
>
>        int                     n_events;
> @@ -1010,6 +1011,7 @@ static int x86_pmu_start(struct perf_event *event)
>        x86_perf_event_set_period(event);
>        cpuc->events[idx] = event;
>        __set_bit(idx, cpuc->active_mask);
> +       __set_bit(idx, cpuc->running);
>        x86_pmu.enable(event);
>        perf_event_update_userpage(event);
>
> @@ -1141,8 +1143,17 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
>        cpuc = &__get_cpu_var(cpu_hw_events);
>
>        for (idx = 0; idx < x86_pmu.num_counters; idx++) {
> -               if (!test_bit(idx, cpuc->active_mask))
> +               if (!test_bit(idx, cpuc->active_mask)) {
> +                       if (__test_and_clear_bit(idx, cpuc->running))
> +                               /*
> +                                * Though we deactivated the counter
> +                                * some cpus might still deliver
> +                                * spurious interrupts. Catching them
> +                                * here.
> +                                */
> +                               handled++;
>                        continue;
> +               }
>
>                event = cpuc->events[idx];
>                hwc = &event->hw;
> --
> 1.7.2.2
>
>
>
>
>
> --
> Advanced Micro Devices, Inc.
> Operating System Research Center
>
>

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Robert Richter]

On 17.09.10 09:06:09, Stephane Eranian wrote:
> Robert.
> 
> Does it mean that with this patch, we don't need Don's back-to-back NMI patch
> anymore?

No, both fix separate things.

Don's patch is about this problem on Intel systems, where ack'ing the
status late may cause empty nmis:

 http://lkml.org/lkml/2010/8/25/124

My patch fixes a problem that after disabling a counter, it could
still cause a late NMI. It was observed on AMD systems, but may also
be valid for Intel systems.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Don Zickus]

On Wed, Sep 15, 2010 at 06:20:34PM +0200, Robert Richter wrote:
> On 14.09.10 19:41:32, Robert Richter wrote:
> > I found the reason why we get the unknown nmi. For some reason
> > cpuc->active_mask in x86_pmu_handle_irq() is zero. Thus, no counters
> > are handled when we get an nmi. It seems there is somewhere a race
> > accessing the active_mask. So far I don't have a fix available.
> > Changing x86_pmu_stop() did not help:
> 
> The patch below for tip/perf/urgent fixes this.
> 
> -Robert

I was able to duplicate the problem and can confirm this patch fixes the
issue for me.  I tried poking around (similar to things Robert probably
did) and had no luck.  Something just doesn't make sense, but I guess for
now this patch is good enough for me. :-)

Cheers,
Don

> 
> From 4206a086f5b37efc1b4d94f1d90b55802b299ca0 Mon Sep 17 00:00:00 2001
> From: Robert Richter <robert.richter@amd.com>
> Date: Wed, 15 Sep 2010 16:12:59 +0200
> Subject: [PATCH] perf, x86: catch spurious interrupts after disabling counters
> 
> Some cpus still deliver spurious interrupts after disabling a counter.
> This caused 'undelivered NMI' messages. This patch fixes this.
> 
> Signed-off-by: Robert Richter <robert.richter@amd.com>
> ---
>  arch/x86/kernel/cpu/perf_event.c |   13 ++++++++++++-
>  1 files changed, 12 insertions(+), 1 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
> index 3efdf28..df7aabd 100644
> --- a/arch/x86/kernel/cpu/perf_event.c
> +++ b/arch/x86/kernel/cpu/perf_event.c
> @@ -102,6 +102,7 @@ struct cpu_hw_events {
>  	 */
>  	struct perf_event	*events[X86_PMC_IDX_MAX]; /* in counter order */
>  	unsigned long		active_mask[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
> +	unsigned long		running[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
>  	int			enabled;
>  
>  	int			n_events;
> @@ -1010,6 +1011,7 @@ static int x86_pmu_start(struct perf_event *event)
>  	x86_perf_event_set_period(event);
>  	cpuc->events[idx] = event;
>  	__set_bit(idx, cpuc->active_mask);
> +	__set_bit(idx, cpuc->running);
>  	x86_pmu.enable(event);
>  	perf_event_update_userpage(event);
>  
> @@ -1141,8 +1143,17 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
>  	cpuc = &__get_cpu_var(cpu_hw_events);
>  
>  	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
> -		if (!test_bit(idx, cpuc->active_mask))
> +		if (!test_bit(idx, cpuc->active_mask)) {
> +			if (__test_and_clear_bit(idx, cpuc->running))
> +				/*
> +				 * Though we deactivated the counter
> +				 * some cpus might still deliver
> +				 * spurious interrupts. Catching them
> +				 * here.
> +				 */
> +				handled++;
>  			continue;
> +		}
>  
>  		event = cpuc->events[idx];
>  		hwc = &event->hw;
> -- 
> 1.7.2.2
> 
> 
> 
> 
> 
> -- 
> Advanced Micro Devices, Inc.
> Operating System Research Center
> 

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Don Zickus]

On Thu, Sep 23, 2010 at 08:02:34PM -0400, Don Zickus wrote:
> On Wed, Sep 15, 2010 at 06:20:34PM +0200, Robert Richter wrote:
> > On 14.09.10 19:41:32, Robert Richter wrote:
> > > I found the reason why we get the unknown nmi. For some reason
> > > cpuc->active_mask in x86_pmu_handle_irq() is zero. Thus, no counters
> > > are handled when we get an nmi. It seems there is somewhere a race
> > > accessing the active_mask. So far I don't have a fix available.
> > > Changing x86_pmu_stop() did not help:
> > 
> > The patch below for tip/perf/urgent fixes this.
> > 
> > -Robert
> 
> I was able to duplicate the problem and can confirm this patch fixes the
> issue for me.  I tried poking around (similar to things Robert probably
> did) and had no luck.  Something just doesn't make sense, but I guess for
> now this patch is good enough for me. :-)

Ah ha!  I figured out what the problem was, need to disable the pmu while
processing the nmi. :-)   Finally something simple in this crazy unknown
NMI spree.

Oh yeah and trace_printk is now my new favorite tool!

From: Don Zickus <dzickus@redhat.com>
Date: Thu, 23 Sep 2010 22:52:09 -0400
Subject: [PATCH] x86, perf: disable pmu from counting when processing irq

On certain AMD and Intel machines, the pmu was left enabled
while the counters were reset during handling of the NMI.
After the counters are reset, code continues to process an
overflow.  During this time another counter overflow interrupt
could happen because the counter is still ticking.  This leads to
an unknown NMI.

static int x86_pmu_handle_irq(struct pt_regs *regs)
{

<snip..>

        for (idx = 0; idx < x86_pmu.num_counters; idx++) {
                if (!test_bit(idx, cpuc->active_mask))
                        continue;

<snip..>

counter reset--> if (!x86_perf_event_set_period(event))
                        continue;

still ticking--> if (perf_event_overflow(event, 1, &data, regs))
<boom overflow>
stopped here -->        x86_pmu_stop(event);

The way to solve this is to disable the pmu while processing the
overflows and re-enable when done.

Signed-off-by: Don Zickus <dzickus@redhat.com>
---
 arch/x86/kernel/cpu/perf_event.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index 48c6d8d..d4fe95d 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1158,6 +1158,8 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 
 	cpuc = &__get_cpu_var(cpu_hw_events);
 
+	x86_pmu_disable_all();
+
 	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
 		if (!test_bit(idx, cpuc->active_mask))
 			continue;
@@ -1182,6 +1184,8 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 			x86_pmu_stop(event, 0);
 	}
 
+	x86_pmu_enable_all(0);
+
 	if (handled)
 		inc_irq_stat(apic_perf_irqs);
 
-- 
1.7.2.3

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Robert Richter]

On 23.09.10 23:18:34, Don Zickus wrote:

> > I was able to duplicate the problem and can confirm this patch fixes the
> > issue for me.  I tried poking around (similar to things Robert probably
> > did) and had no luck.  Something just doesn't make sense, but I guess for
> > now this patch is good enough for me. :-)
> 
> Ah ha!  I figured out what the problem was, need to disable the pmu while
> processing the nmi. :-)   Finally something simple in this crazy unknown
> NMI spree.
> 
> Oh yeah and trace_printk is now my new favorite tool!
> 
> From: Don Zickus <dzickus@redhat.com>
> Date: Thu, 23 Sep 2010 22:52:09 -0400
> Subject: [PATCH] x86, perf: disable pmu from counting when processing irq
> 
> On certain AMD and Intel machines, the pmu was left enabled
> while the counters were reset during handling of the NMI.
> After the counters are reset, code continues to process an
> overflow.  During this time another counter overflow interrupt
> could happen because the counter is still ticking.  This leads to
> an unknown NMI.

I don't like the approach of disabling all counters in the nmi
handler. First, it stops counting and thus may falsify
measurement. Second, it introduces much overhead doing a rd-/wrmsrl()
for each counter.

Does your patch below solve something that my patch doesn't?

Btw, Ingo, my patch should be applied to tip/perf/urgent as it fixes
the regression you discovered on AMD systems.

Thanks,

-Robert

> 
> static int x86_pmu_handle_irq(struct pt_regs *regs)
> {
> 
> <snip..>
> 
>         for (idx = 0; idx < x86_pmu.num_counters; idx++) {
>                 if (!test_bit(idx, cpuc->active_mask))
>                         continue;
> 
> <snip..>
> 
> counter reset--> if (!x86_perf_event_set_period(event))
>                         continue;
> 
> still ticking--> if (perf_event_overflow(event, 1, &data, regs))
> <boom overflow>
> stopped here -->        x86_pmu_stop(event);
> 
> The way to solve this is to disable the pmu while processing the
> overflows and re-enable when done.
> 
> Signed-off-by: Don Zickus <dzickus@redhat.com>
> ---
>  arch/x86/kernel/cpu/perf_event.c |    4 ++++
>  1 files changed, 4 insertions(+), 0 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
> index 48c6d8d..d4fe95d 100644
> --- a/arch/x86/kernel/cpu/perf_event.c
> +++ b/arch/x86/kernel/cpu/perf_event.c
> @@ -1158,6 +1158,8 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
>  
>  	cpuc = &__get_cpu_var(cpu_hw_events);
>  
> +	x86_pmu_disable_all();
> +
>  	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
>  		if (!test_bit(idx, cpuc->active_mask))
>  			continue;
> @@ -1182,6 +1184,8 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
>  			x86_pmu_stop(event, 0);
>  	}
>  
> +	x86_pmu_enable_all(0);
> +
>  	if (handled)
>  		inc_irq_stat(apic_perf_irqs);
>  
> -- 
> 1.7.2.3
> 
> 

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Stephane Eranian]

On Fri, Sep 24, 2010 at 12:03 PM, Robert Richter <robert.richter@amd.com> wrote:
> On 23.09.10 23:18:34, Don Zickus wrote:
>
>> > I was able to duplicate the problem and can confirm this patch fixes the
>> > issue for me.  I tried poking around (similar to things Robert probably
>> > did) and had no luck.  Something just doesn't make sense, but I guess for
>> > now this patch is good enough for me. :-)
>>
>> Ah ha!  I figured out what the problem was, need to disable the pmu while
>> processing the nmi. :-)   Finally something simple in this crazy unknown
>> NMI spree.
>>
>> Oh yeah and trace_printk is now my new favorite tool!
>>
>> From: Don Zickus <dzickus@redhat.com>
>> Date: Thu, 23 Sep 2010 22:52:09 -0400
>> Subject: [PATCH] x86, perf: disable pmu from counting when processing irq
>>
>> On certain AMD and Intel machines, the pmu was left enabled
>> while the counters were reset during handling of the NMI.
>> After the counters are reset, code continues to process an
>> overflow.  During this time another counter overflow interrupt
>> could happen because the counter is still ticking.  This leads to
>> an unknown NMI.
>
> I don't like the approach of disabling all counters in the nmi
> handler. First, it stops counting and thus may falsify
> measurement. Second, it introduces much overhead doing a rd-/wrmsrl()
> for each counter.
>
But that's exactly what is going on the Intel side. PMU is stopped on interrupt.
An argument for this is that you don't necessarily want to monitor across
the PMU handler, i.e., the overhead you introduce.

> Does your patch below solve something that my patch doesn't?
>
> Btw, Ingo, my patch should be applied to tip/perf/urgent as it fixes
> the regression you discovered on AMD systems.
>
> Thanks,
>
> -Robert
>
>>
>> static int x86_pmu_handle_irq(struct pt_regs *regs)
>> {
>>
>> <snip..>
>>
>>         for (idx = 0; idx < x86_pmu.num_counters; idx++) {
>>                 if (!test_bit(idx, cpuc->active_mask))
>>                         continue;
>>
>> <snip..>
>>
>> counter reset--> if (!x86_perf_event_set_period(event))
>>                         continue;
>>
>> still ticking--> if (perf_event_overflow(event, 1, &data, regs))
>> <boom overflow>
>> stopped here -->        x86_pmu_stop(event);
>>
>> The way to solve this is to disable the pmu while processing the
>> overflows and re-enable when done.
>>
>> Signed-off-by: Don Zickus <dzickus@redhat.com>
>> ---
>>  arch/x86/kernel/cpu/perf_event.c |    4 ++++
>>  1 files changed, 4 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
>> index 48c6d8d..d4fe95d 100644
>> --- a/arch/x86/kernel/cpu/perf_event.c
>> +++ b/arch/x86/kernel/cpu/perf_event.c
>> @@ -1158,6 +1158,8 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
>>
>>       cpuc = &__get_cpu_var(cpu_hw_events);
>>
>> +     x86_pmu_disable_all();
>> +
>>       for (idx = 0; idx < x86_pmu.num_counters; idx++) {
>>               if (!test_bit(idx, cpuc->active_mask))
>>                       continue;
>> @@ -1182,6 +1184,8 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
>>                       x86_pmu_stop(event, 0);
>>       }
>>
>> +     x86_pmu_enable_all(0);
>> +
>>       if (handled)
>>               inc_irq_stat(apic_perf_irqs);
>>
>> --
>> 1.7.2.3
>>
>>
>
> --
> Advanced Micro Devices, Inc.
> Operating System Research Center
>
>

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Peter Zijlstra]

On Fri, 2010-09-24 at 15:38 +0200, Stephane Eranian wrote:
> > I don't like the approach of disabling all counters in the nmi
> > handler. First, it stops counting and thus may falsify
> > measurement. Second, it introduces much overhead doing a rd-/wrmsrl()
> > for each counter.
> >
> But that's exactly what is going on the Intel side. PMU is stopped on interrupt.
> An argument for this is that you don't necessarily want to monitor across
> the PMU handler, i.e., the overhead you introduce.

Right, its really a question of what you want to measure. I prefer not
to measure the measuring itself, things are hard enough to interpret
already.

Then again, the Intel stuff has a real handy way to disable the whole
PMU, unlike the AMD bits where you need to iterate each counter
individually.

Re: [PATCH] perf, x86: catch spurious interrupts after disabling counters

[Don Zickus]

On Fri, Sep 24, 2010 at 12:03:45PM +0200, Robert Richter wrote:
> On 23.09.10 23:18:34, Don Zickus wrote:
> 
> > > I was able to duplicate the problem and can confirm this patch fixes the
> > > issue for me.  I tried poking around (similar to things Robert probably
> > > did) and had no luck.  Something just doesn't make sense, but I guess for
> > > now this patch is good enough for me. :-)
> > 
> > Ah ha!  I figured out what the problem was, need to disable the pmu while
> > processing the nmi. :-)   Finally something simple in this crazy unknown
> > NMI spree.
> > 
> > Oh yeah and trace_printk is now my new favorite tool!
> > 
> > From: Don Zickus <dzickus@redhat.com>
> > Date: Thu, 23 Sep 2010 22:52:09 -0400
> > Subject: [PATCH] x86, perf: disable pmu from counting when processing irq
> > 
> > On certain AMD and Intel machines, the pmu was left enabled
> > while the counters were reset during handling of the NMI.
> > After the counters are reset, code continues to process an
> > overflow.  During this time another counter overflow interrupt
> > could happen because the counter is still ticking.  This leads to
> > an unknown NMI.
> 
> I don't like the approach of disabling all counters in the nmi
> handler. First, it stops counting and thus may falsify
> measurement. Second, it introduces much overhead doing a rd-/wrmsrl()
> for each counter.

Ok. good points.  Though we may have to re-visit the perf_event_intel.c
code as my patch was based on 'well they did it there, we can do it here'
approach.

> 
> Does your patch below solve something that my patch doesn't?

Well, two things I was trying to solve with your approach is the extra NMI
that is generated, and the fact that you may falsely eat an unknown NMI
(because you don't clear cpuc->running except in the special case).

Yeah, I know the heurestics from your patch a couple of weeks ago will
make the case of falsely eating an unknown NMI next to impossible.  But I
was trying to limit the number seen.

For example, the back-to-back nmi problem we dealt with a couple of weeks
ago, only had a 'special case' that had to be dealt with every once in a
while (depending on the level of perf activity).  Now I can see unknown
NMIs every time there is a pmu_stop() issued.  I was trying to figure out
a way to cut down on that noise.

I came up with a couple of approaches but both involve another rdmsrl in
the handle_irq path.  I thought I would toss these ideas out for
conversation.

the first approach gets rid of the extra nmi by waiting until the end to
re-write the counter, but adds a second rdmsrl to resync the period in the
case where pmu_stop is not called.


diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index 48c6d8d..1642f48 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1175,11 +1175,22 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 		handled++;
 		data.period	= event->hw.last_period;
 
-		if (!x86_perf_event_set_period(event))
-			continue;
+		/*
+		 * if period is over, process the overflow
+		 * before reseting the counter, otherwise
+		 * a new overflow could occur before the 
+		 * event is stopped
+		 */
+		if (local64_read(&hwc->period_left) <= 0) {
+			if (perf_event_overflow(event, 1, &data, regs)) {
+				x86_pmu_stop(event, 0);
+				continue;
+			}
+			/* if the overflow doesn't stop the event, resync */
+			x86_perf_event_update(event);
+		}
 
-		if (perf_event_overflow(event, 1, &data, regs))
-			x86_pmu_stop(event, 0);
+		x86_perf_event_set_period(event);
 	}
 
 	if (handled)

and the second approach, accepts the fact that we will get another unknown
NMI but check for it first (after stopping) and clear the cpuc->running
bit if we didn't overflow yet.

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index 18c8856..bfb05da 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1189,8 +1189,12 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 		if (!x86_perf_event_set_period(event))
 			continue;
 
-		if (perf_event_overflow(event, 1, &data, regs))
+		if (perf_event_overflow(event, 1, &data, regs)) {
 			x86_pmu_stop(event, 0);
+			rdmsrl(hwc->event_base + idx, val);
+			if (val & (1ULL << (x86_pmu.cntval_bits - 1)))
+				__clear_bit(idx, cpuc->running);
+		}
 	}
 
 	if (handled)


After dealing with the Intel folks on another thread about how they are
finding more ways to detect and report hardware errors with NMIs, that I was
getting nervous about generating so many false NMIs and accidentally
eating a real one.

Cheers,
Don

[tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[tip-bot for Robert Richter]

Commit-ID:  63e6be6d98e1a2bcdca86872b67052e51ab6afa1
Gitweb:     http://git.kernel.org/tip/63e6be6d98e1a2bcdca86872b67052e51ab6afa1
Author:     Robert Richter <robert.richter@amd.com>
AuthorDate: Wed, 15 Sep 2010 18:20:34 +0200
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 24 Sep 2010 12:21:41 +0200

perf, x86: Catch spurious interrupts after disabling counters

Some cpus still deliver spurious interrupts after disabling a
counter. This caused 'undelivered NMI' messages. This patch
fixes this. Introduced by:

  4177c42: perf, x86: Try to handle unknown nmis with an enabled PMU

Reported-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Robert Richter <robert.richter@amd.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: gorcunov@gmail.com <gorcunov@gmail.com>
Cc: fweisbec@gmail.com <fweisbec@gmail.com>
Cc: ying.huang@intel.com <ying.huang@intel.com>
Cc: ming.m.lin@intel.com <ming.m.lin@intel.com>
Cc: yinghai@kernel.org <yinghai@kernel.org>
Cc: andi@firstfloor.org <andi@firstfloor.org>
Cc: eranian@google.com <eranian@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
LKML-Reference: <20100915162034.GO13563@erda.amd.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 arch/x86/kernel/cpu/perf_event.c |   12 +++++++++++-
 1 files changed, 11 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index 3efdf28..03a5b03 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -102,6 +102,7 @@ struct cpu_hw_events {
 	 */
 	struct perf_event	*events[X86_PMC_IDX_MAX]; /* in counter order */
 	unsigned long		active_mask[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
+	unsigned long		running[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
 	int			enabled;
 
 	int			n_events;
@@ -1010,6 +1011,7 @@ static int x86_pmu_start(struct perf_event *event)
 	x86_perf_event_set_period(event);
 	cpuc->events[idx] = event;
 	__set_bit(idx, cpuc->active_mask);
+	__set_bit(idx, cpuc->running);
 	x86_pmu.enable(event);
 	perf_event_update_userpage(event);
 
@@ -1141,8 +1143,16 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 	cpuc = &__get_cpu_var(cpu_hw_events);
 
 	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
-		if (!test_bit(idx, cpuc->active_mask))
+		if (!test_bit(idx, cpuc->active_mask)) {
+			/*
+			 * Though we deactivated the counter some cpus
+			 * might still deliver spurious interrupts still
+			 * in flight. Catch them:
+			 */
+			if (__test_and_clear_bit(idx, cpuc->running))
+				handled++;
 			continue;
+		}
 
 		event = cpuc->events[idx];
 		hwc = &event->hw;

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Stephane Eranian]

Robert,

You've applied the fix only to the generic X86 interrupt handler
which is currently used by AMD64 processors.

It seems to me that this "in-flight interrupt after disable" problem
may also happen on Intel and should therefore also be added
to intel_pmu_handle_irq(). Don't you think so?


On Fri, Sep 24, 2010 at 12:41 PM, tip-bot for Robert Richter
<robert.richter@amd.com> wrote:
>
> Commit-ID:  63e6be6d98e1a2bcdca86872b67052e51ab6afa1
> Gitweb:     http://git.kernel.org/tip/63e6be6d98e1a2bcdca86872b67052e51ab6afa1
> Author:     Robert Richter <robert.richter@amd.com>
> AuthorDate: Wed, 15 Sep 2010 18:20:34 +0200
> Committer:  Ingo Molnar <mingo@elte.hu>
> CommitDate: Fri, 24 Sep 2010 12:21:41 +0200
>
> perf, x86: Catch spurious interrupts after disabling counters
>
> Some cpus still deliver spurious interrupts after disabling a
> counter. This caused 'undelivered NMI' messages. This patch
> fixes this. Introduced by:
>
>  4177c42: perf, x86: Try to handle unknown nmis with an enabled PMU
>
> Reported-by: Ingo Molnar <mingo@elte.hu>
> Signed-off-by: Robert Richter <robert.richter@amd.com>
> Cc: Don Zickus <dzickus@redhat.com>
> Cc: gorcunov@gmail.com <gorcunov@gmail.com>
> Cc: fweisbec@gmail.com <fweisbec@gmail.com>
> Cc: ying.huang@intel.com <ying.huang@intel.com>
> Cc: ming.m.lin@intel.com <ming.m.lin@intel.com>
> Cc: yinghai@kernel.org <yinghai@kernel.org>
> Cc: andi@firstfloor.org <andi@firstfloor.org>
> Cc: eranian@google.com <eranian@google.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> LKML-Reference: <20100915162034.GO13563@erda.amd.com>
> Signed-off-by: Ingo Molnar <mingo@elte.hu>
> ---
>  arch/x86/kernel/cpu/perf_event.c |   12 +++++++++++-
>  1 files changed, 11 insertions(+), 1 deletions(-)
>
> diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
> index 3efdf28..03a5b03 100644
> --- a/arch/x86/kernel/cpu/perf_event.c
> +++ b/arch/x86/kernel/cpu/perf_event.c
> @@ -102,6 +102,7 @@ struct cpu_hw_events {
>         */
>        struct perf_event       *events[X86_PMC_IDX_MAX]; /* in counter order */
>        unsigned long           active_mask[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
> +       unsigned long           running[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
>        int                     enabled;
>
>        int                     n_events;
> @@ -1010,6 +1011,7 @@ static int x86_pmu_start(struct perf_event *event)
>        x86_perf_event_set_period(event);
>        cpuc->events[idx] = event;
>        __set_bit(idx, cpuc->active_mask);
> +       __set_bit(idx, cpuc->running);
>        x86_pmu.enable(event);
>        perf_event_update_userpage(event);
>
> @@ -1141,8 +1143,16 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
>        cpuc = &__get_cpu_var(cpu_hw_events);
>
>        for (idx = 0; idx < x86_pmu.num_counters; idx++) {
> -               if (!test_bit(idx, cpuc->active_mask))
> +               if (!test_bit(idx, cpuc->active_mask)) {
> +                       /*
> +                        * Though we deactivated the counter some cpus
> +                        * might still deliver spurious interrupts still
> +                        * in flight. Catch them:
> +                        */
> +                       if (__test_and_clear_bit(idx, cpuc->running))
> +                               handled++;
>                        continue;
> +               }
>
>                event = cpuc->events[idx];
>                hwc = &event->hw;

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Robert Richter]

Stephane,

On 29.09.10 08:26:41, Stephane Eranian wrote:
> You've applied the fix only to the generic X86 interrupt handler
> which is currently used by AMD64 processors.

(... and P4).

> It seems to me that this "in-flight interrupt after disable" problem
> may also happen on Intel and should therefore also be added
> to intel_pmu_handle_irq(). Don't you think so?

It only happens if the active_mask is used for checking single
counters for overflows.

Systems with Intel Architectural Perfmon use the status mask msr to
determine which counter overflowed. In intel_pmu_handle_irq() the
handled counter is incremented in this case even if the counter is not
active, so everything should be fine here.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Robert Richter]

On 29.09.10 14:53:01, Robert Richter wrote:
> Stephane,
> 
> On 29.09.10 08:26:41, Stephane Eranian wrote:
> > You've applied the fix only to the generic X86 interrupt handler
> > which is currently used by AMD64 processors.
> 
> (... and P4).
> 
> > It seems to me that this "in-flight interrupt after disable" problem
> > may also happen on Intel and should therefore also be added
> > to intel_pmu_handle_irq(). Don't you think so?
> 
> It only happens if the active_mask is used for checking single
> counters for overflows.
> 
> Systems with Intel Architectural Perfmon use the status mask msr to
> determine which counter overflowed. In intel_pmu_handle_irq() the
> handled counter is incremented in this case even if the counter is not

I mean handled count (variable 'handled').

> active, so everything should be fine here.
> 
> -Robert
> 
> -- 
> Advanced Micro Devices, Inc.
> Operating System Research Center

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Stephane Eranian]

On Wed, Sep 29, 2010 at 2:54 PM, Robert Richter <robert.richter@amd.com> wrote:
> On 29.09.10 14:53:01, Robert Richter wrote:
>> Stephane,
>>
>> On 29.09.10 08:26:41, Stephane Eranian wrote:
>> > You've applied the fix only to the generic X86 interrupt handler
>> > which is currently used by AMD64 processors.
>>
>> (... and P4).
>>
>> > It seems to me that this "in-flight interrupt after disable" problem
>> > may also happen on Intel and should therefore also be added
>> > to intel_pmu_handle_irq(). Don't you think so?
>>
>> It only happens if the active_mask is used for checking single
>> counters for overflows.
>>
On Intel there is a double-check. You obtain the overflow mask
from HW. For each bit set, you verify it corresponds to an active
event. If not, you ignore:

for_each_set_bit(bit, (unsigned long *)&status, X86_PMC_IDX_MAX) {
                struct perf_event *event = cpuc->events[bit];

                handled++;

                if (!test_bit(bit, cpuc->active_mask))
                    continue;
}

I think the logic is similar. What makes the difference, it seems, is that
handled is incremented unconditionally if the ovfl_mask says it has
an overflow, i.e., before active_mask is checked.

On Westmere, we've seen situations where the overflow mask and active
mask did not agree.

On counter disable, the overflow mask bit is not cleared, thus one may iterate
in the loop and fail the active_mask. But handled would be incremented in that
case, so that would behave like in your patch.




>> Systems with Intel Architectural Perfmon use the status mask msr to
>> determine which counter overflowed. In intel_pmu_handle_irq() the
>> handled counter is incremented in this case even if the counter is not
>
> I mean handled count (variable 'handled').
>
>> active, so everything should be fine here.
>>
>> -Robert
>>
>> --
>> Advanced Micro Devices, Inc.
>> Operating System Research Center
>
> --
> Advanced Micro Devices, Inc.
> Operating System Research Center
>
>

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Stephane Eranian]

On Wed, Sep 29, 2010 at 3:13 PM, Stephane Eranian <eranian@google.com> wrote:
> On Wed, Sep 29, 2010 at 2:54 PM, Robert Richter <robert.richter@amd.com> wrote:
>> On 29.09.10 14:53:01, Robert Richter wrote:
>>> Stephane,
>>>
>>> On 29.09.10 08:26:41, Stephane Eranian wrote:
>>> > You've applied the fix only to the generic X86 interrupt handler
>>> > which is currently used by AMD64 processors.
>>>
>>> (... and P4).
>>>
Well, in tip-x86, I don't see your fix in p4_pmu_handle_irq(). Is
that pending?

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Robert Richter]

On 29.09.10 09:28:56, Stephane Eranian wrote:
> On Wed, Sep 29, 2010 at 3:13 PM, Stephane Eranian <eranian@google.com> wrote:
> > On Wed, Sep 29, 2010 at 2:54 PM, Robert Richter <robert.richter@amd.com> wrote:
> >> On 29.09.10 14:53:01, Robert Richter wrote:
> >>> Stephane,
> >>>
> >>> On 29.09.10 08:26:41, Stephane Eranian wrote:
> >>> > You've applied the fix only to the generic X86 interrupt handler
> >>> > which is currently used by AMD64 processors.
> >>>
> >>> (... and P4).
> >>>

> Well, in tip-x86, I don't see your fix in p4_pmu_handle_irq(). Is
> that pending?

Right, I wasn't remembering correctly, it was P6 and core. And yes, P4
requires the fix. Will send a patch for this.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Robert Richter]

On 29.09.10 17:01:40, Robert Richter wrote:
> Right, I wasn't remembering correctly, it was P6 and core. And yes, P4
> requires the fix. Will send a patch for this.

Ok, Cyrill is sending the patch.

Thanks,

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Cyrill Gorcunov]

On Wed, Sep 29, 2010 at 05:12:53PM +0200, Robert Richter wrote:
> On 29.09.10 17:01:40, Robert Richter wrote:
> > Right, I wasn't remembering correctly, it was P6 and core. And yes, P4
> > requires the fix. Will send a patch for this.
> 
> Ok, Cyrill is sending the patch.
> 
> Thanks,
> 
> -Robert
>

Managed to implement it earlier then I thought.

  Cyrill
---
Subject: [PATCH -tip] perf, x86: Handle in flight IRQs on P4 platform

Stephane reported we've missed to guard P4 platform
against spurious in-flight performance IRQs. Fix it.

[ the patch is a complement to commit 63e6be6d98e1 ]

Reported-by: Stephane Eranian <eranian@google.com>
Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
CC: Robert Richter <robert.richter@amd.com>
CC: Lin Ming <ming.m.lin@intel.com>
---
 arch/x86/kernel/cpu/perf_event_p4.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

Index: linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
=====================================================================
--- linux-2.6.git.orig/arch/x86/kernel/cpu/perf_event_p4.c
+++ linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
@@ -904,8 +904,12 @@ static int p4_pmu_handle_irq(struct pt_r
 	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
 		int overflow;
 
-		if (!test_bit(idx, cpuc->active_mask))
+		if (!test_bit(idx, cpuc->active_mask)) {
+			/* catch in-flight IRQs */
+			if (__test_and_clear_bit(idx, cpuc->running))
+				handled++;
 			continue;
+		}
 
 		event = cpuc->events[idx];
 		hwc = &event->hw;

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Stephane Eranian]

Robert,

There is something else bothering me with cpuc->running.

It is not reset outside of the interrupt handler. So what if
event scheduling shuffles things around and an event is
moved somewhere else. Don't you need to clear the
cpuc->running[idx] for the old counter index?


On Wed, Sep 29, 2010 at 5:27 PM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> On Wed, Sep 29, 2010 at 05:12:53PM +0200, Robert Richter wrote:
>> On 29.09.10 17:01:40, Robert Richter wrote:
>> > Right, I wasn't remembering correctly, it was P6 and core. And yes, P4
>> > requires the fix. Will send a patch for this.
>>
>> Ok, Cyrill is sending the patch.
>>
>> Thanks,
>>
>> -Robert
>>
>
> Managed to implement it earlier then I thought.
>
>  Cyrill
> ---
> Subject: [PATCH -tip] perf, x86: Handle in flight IRQs on P4 platform
>
> Stephane reported we've missed to guard P4 platform
> against spurious in-flight performance IRQs. Fix it.
>
> [ the patch is a complement to commit 63e6be6d98e1 ]
>
> Reported-by: Stephane Eranian <eranian@google.com>
> Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
> CC: Robert Richter <robert.richter@amd.com>
> CC: Lin Ming <ming.m.lin@intel.com>
> ---
>  arch/x86/kernel/cpu/perf_event_p4.c |    6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> Index: linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
> =====================================================================
> --- linux-2.6.git.orig/arch/x86/kernel/cpu/perf_event_p4.c
> +++ linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
> @@ -904,8 +904,12 @@ static int p4_pmu_handle_irq(struct pt_r
>        for (idx = 0; idx < x86_pmu.num_counters; idx++) {
>                int overflow;
>
> -               if (!test_bit(idx, cpuc->active_mask))
> +               if (!test_bit(idx, cpuc->active_mask)) {
> +                       /* catch in-flight IRQs */
> +                       if (__test_and_clear_bit(idx, cpuc->running))
> +                               handled++;
>                        continue;
> +               }
>
>                event = cpuc->events[idx];
>                hwc = &event->hw;
>

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Cyrill Gorcunov]

On Wed, Sep 29, 2010 at 05:33:07PM +0200, Stephane Eranian wrote:
> Robert,
> 
> There is something else bothering me with cpuc->running.
> 
> It is not reset outside of the interrupt handler. So what if
> event scheduling shuffles things around and an event is
> moved somewhere else. Don't you need to clear the
> cpuc->running[idx] for the old counter index?
> 
> 

Both bitmasks are set and test with same index though it might
be a bit obscure scheme (we could be clearing this bit in
x86_pmu_stop but it just a wasting cycles).

Btw, since x86 architectural and p4 are using same tests for
running I presume better to have some helper rather then
open coded pile?

Cyrill

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Cyrill Gorcunov]

On Wed, Sep 29, 2010 at 07:45:28PM +0400, Cyrill Gorcunov wrote:
... 
> Btw, since x86 architectural and p4 are using same tests for
> running I presume better to have some helper rather then
> open coded pile?
> 
> Cyrill

Something like this (with some better comment in middle)?

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index 6526a86..73a07fb 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1147,6 +1147,16 @@ static void x86_pmu_del(struct perf_event *event, int flags)
 	perf_event_update_userpage(event);
 }
 
+static inline int x86_pmu_spurious_irq(int idx, struct cpu_hw_events *cpuc)
+{
+	/*
+	 * Though we deactivated the counter some cpus
+	 * might still deliver spurious interrupts still
+	 * in flight. Catch them:
+	 */
+	return __test_and_clear_bit(idx, cpuc->running);
+}
+
 static int x86_pmu_handle_irq(struct pt_regs *regs)
 {
 	struct perf_sample_data data;
@@ -1162,13 +1172,7 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 
 	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
 		if (!test_bit(idx, cpuc->active_mask)) {
-			/*
-			 * Though we deactivated the counter some cpus
-			 * might still deliver spurious interrupts still
-			 * in flight. Catch them:
-			 */
-			if (__test_and_clear_bit(idx, cpuc->running))
-				handled++;
+			handled += x86_pmu_spurious_irq(idx, cpuc);
 			continue;
 		}
 
diff --git a/arch/x86/kernel/cpu/perf_event_p4.c b/arch/x86/kernel/cpu/perf_event_p4.c
index c70c878..524325c 100644
--- a/arch/x86/kernel/cpu/perf_event_p4.c
+++ b/arch/x86/kernel/cpu/perf_event_p4.c
@@ -904,8 +904,10 @@ static int p4_pmu_handle_irq(struct pt_regs *regs)
 	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
 		int overflow;
 
-		if (!test_bit(idx, cpuc->active_mask))
+		if (!test_bit(idx, cpuc->active_mask)) {
+			handled += x86_pmu_spurious_irq(idx, cpuc);
 			continue;
+		}
 
 		event = cpuc->events[idx];
 		hwc = &event->hw;

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Robert Richter]

On 29.09.10 11:51:02, Cyrill Gorcunov wrote:
> On Wed, Sep 29, 2010 at 07:45:28PM +0400, Cyrill Gorcunov wrote:
> ... 
> > Btw, since x86 architectural and p4 are using same tests for
> > running I presume better to have some helper rather then
> > open coded pile?

I rather tend to merge the x86 and p4 handler, the code is almost the
same.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Cyrill Gorcunov]

On Wed, Sep 29, 2010 at 06:32:46PM +0200, Robert Richter wrote:
> On 29.09.10 11:51:02, Cyrill Gorcunov wrote:
> > On Wed, Sep 29, 2010 at 07:45:28PM +0400, Cyrill Gorcunov wrote:
> > ... 
> > > Btw, since x86 architectural and p4 are using same tests for
> > > running I presume better to have some helper rather then
> > > open coded pile?
> 
> I rather tend to merge the x86 and p4 handler, the code is almost the
> same.
> 

There are still some difference which would require if() in code depending
on which real cpu code is running, that is why I didn't merge it in a first
place. And branch misprediction penalty might worth to not merge such
critical path. On the other hands perhaps we might look into "alternatives"
and just nopify some calls early in cpu initialization.

> -Robert
> 
> -- 
> Advanced Micro Devices, Inc.
> Operating System Research Center
> 
  Cyrill

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Stephane Eranian]

On Wed, Sep 29, 2010 at 5:45 PM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> On Wed, Sep 29, 2010 at 05:33:07PM +0200, Stephane Eranian wrote:
>> Robert,
>>
>> There is something else bothering me with cpuc->running.
>>
>> It is not reset outside of the interrupt handler. So what if
>> event scheduling shuffles things around and an event is
>> moved somewhere else. Don't you need to clear the
>> cpuc->running[idx] for the old counter index?
>>
>>
>
> Both bitmasks are set and test with same index though it might
> be a bit obscure scheme (we could be clearing this bit in
> x86_pmu_stop but it just a wasting cycles).
>
But you cannot clear it in x86_pmu_stop() because otherwise it
turns into active_mask[]. My understanding is that you need
to remember this counter has been active at some point in the
past.

My point is that you cannot keep this around forever. After a
"while" it becomes stale and you have to remove it otherwise
you may wrongly increment handled.

Here is a scenario:

event A -> counter 0, cpuc->running = 0x1 active_mask = 0x1
move A
event A -> counter 1, cpuc->running = 0x3, active_mask = 0x2

No interrupt, we are just counting for a short period.
Then, you get an NMI interrupt, suppose it is not generated
by the PMU, it is destined for another handler.

For i=0, you have (active_mask & 0x1) == 0, but (running & 0x1) == 1,
you mark the interrupt as handled, i.e., you swallow it, the actual
handler never gets it.


> Btw, since x86 architectural and p4 are using same tests for
> running I presume better to have some helper rather then
> open coded pile?
>
> Cyrill
>

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Robert Richter]

On 29.09.10 12:00:35, Stephane Eranian wrote:
> But you cannot clear it in x86_pmu_stop() because otherwise it
> turns into active_mask[]. My understanding is that you need
> to remember this counter has been active at some point in the
> past.
> 
> My point is that you cannot keep this around forever. After a
> "while" it becomes stale and you have to remove it otherwise
> you may wrongly increment handled.

The mask is cleared with the next nmi.

> 
> Here is a scenario:
> 
> event A -> counter 0, cpuc->running = 0x1 active_mask = 0x1
> move A
> event A -> counter 1, cpuc->running = 0x3, active_mask = 0x2
> 
> No interrupt, we are just counting for a short period.
> Then, you get an NMI interrupt, suppose it is not generated
> by the PMU, it is destined for another handler.
> 
> For i=0, you have (active_mask & 0x1) == 0, but (running & 0x1) == 1,
> you mark the interrupt as handled, i.e., you swallow it, the actual
> handler never gets it.

Yes, then changing the counters you will get *one* nmi with 2 handled
counters. This is valid as the disabled counter could generate a
spurious interrupt. But you get (handled == 2) instead of (handled ==
1) which is not much impact. All following nmis have (handled == 1)
then again.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Cyrill Gorcunov]

On Wed, Sep 29, 2010 at 07:09:24PM +0200, Robert Richter wrote:
> On 29.09.10 12:00:35, Stephane Eranian wrote:
> > But you cannot clear it in x86_pmu_stop() because otherwise it
> > turns into active_mask[]. My understanding is that you need

indeed

  Cyrill

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Don Zickus]

On Wed, Sep 29, 2010 at 07:09:24PM +0200, Robert Richter wrote:
> On 29.09.10 12:00:35, Stephane Eranian wrote:
> > Here is a scenario:
> > 
> > event A -> counter 0, cpuc->running = 0x1 active_mask = 0x1
> > move A
> > event A -> counter 1, cpuc->running = 0x3, active_mask = 0x2
> > 
> > No interrupt, we are just counting for a short period.
> > Then, you get an NMI interrupt, suppose it is not generated
> > by the PMU, it is destined for another handler.
> > 
> > For i=0, you have (active_mask & 0x1) == 0, but (running & 0x1) == 1,
> > you mark the interrupt as handled, i.e., you swallow it, the actual
> > handler never gets it.
> 
> Yes, then changing the counters you will get *one* nmi with 2 handled
> counters. This is valid as the disabled counter could generate a
> spurious interrupt. But you get (handled == 2) instead of (handled ==
> 1) which is not much impact. All following nmis have (handled == 1)
> then again.

Robert, 

I think you missed Stephane's point.  Say for example, kgdb is being used
while we are doing stuff with the perf counter (and say kgdb's handler is
a lower priority than perf; which isn't true I know, but let's say):

perf NMI comes in, issues pmu_stop 'cleanly' (meaning no spurious
interrupt).  The cpuc->running bit is never cleared.

kgdb NMI comes in, but the die_chain dictates perf looks at it first.
perf will see that cpuc->active == 0 and cpuc->running == 1 and bump
handled.  Thus returning NOTIFY_STOP.  kgdb never sees the NMI. :-(

Now I sent a patch last week that can prevent that extra NMI from being
generated at the cost of another rdmsrl in the non-pmu_stop cases (which I
will attach below again, obviously P4 would need something similar too).

I think we currently don't see the problems Stephane describes because the
only thing we test that uses NMIs are perf, which also happens to be a low
priority on the die_chain.

But it is an interesting scenario that we should look at more.

Cheers,
Don


diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index 48c6d8d..1642f48 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1175,11 +1175,22 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 		handled++;
 		data.period	= event->hw.last_period;
 
-		if (!x86_perf_event_set_period(event))
-			continue;
+		/*
+		 * if period is over, process the overflow
+		 * before reseting the counter, otherwise
+		 * a new overflow could occur before the 
+		 * event is stopped
+		 */
+		if (local64_read(&hwc->period_left) <= 0) {
+			if (perf_event_overflow(event, 1, &data, regs)) {
+				x86_pmu_stop(event, 0);
+				continue;
+			}
+			/* if the overflow doesn't stop the event, resync */
+			x86_perf_event_update(event);
+		}
 
-		if (perf_event_overflow(event, 1, &data, regs))
-			x86_pmu_stop(event, 0);
+		x86_perf_event_set_period(event);
 	}
 
 	if (handled)

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Stephane Eranian]

On Wed, Sep 29, 2010 at 8:12 PM, Don Zickus <dzickus@redhat.com> wrote:
> On Wed, Sep 29, 2010 at 07:09:24PM +0200, Robert Richter wrote:
>> On 29.09.10 12:00:35, Stephane Eranian wrote:
>> > Here is a scenario:
>> >
>> > event A -> counter 0, cpuc->running = 0x1 active_mask = 0x1
>> > move A
>> > event A -> counter 1, cpuc->running = 0x3, active_mask = 0x2
>> >
>> > No interrupt, we are just counting for a short period.
>> > Then, you get an NMI interrupt, suppose it is not generated
>> > by the PMU, it is destined for another handler.
>> >
>> > For i=0, you have (active_mask & 0x1) == 0, but (running & 0x1) == 1,
>> > you mark the interrupt as handled, i.e., you swallow it, the actual
>> > handler never gets it.
>>
>> Yes, then changing the counters you will get *one* nmi with 2 handled
>> counters. This is valid as the disabled counter could generate a
>> spurious interrupt. But you get (handled == 2) instead of (handled ==
>> 1) which is not much impact. All following nmis have (handled == 1)
>> then again.
>
> Robert,
>
> I think you missed Stephane's point.  Say for example, kgdb is being used
> while we are doing stuff with the perf counter (and say kgdb's handler is
> a lower priority than perf; which isn't true I know, but let's say):
>
Yes, exactly my point. The reality is you cannot afford to have false positive
because you may starve another subsystem from an important notification.

I think it boils down to whether or not we need an error message (Dazed) in
case no subsystem claimed the NMI. If you were to just silently consume the
NMI when no subsystem claims it, then you would not have these issues.

What Don has done is use a heuristic which gets activated when a PMU
interrupt handler signals that more than one counter have overflowed. His
claim is that this situation is likely to trigger back-to-back.

The reason this heuristic works is because it waits until ALL the subsystems
have seen the notification before it declares that the NMI was PMU spurious.
To do that is uses the DIE_NMI_UNKNOWN callchain. Handler on this chain
get call last, after all subsystems have seen the notification once. I believe
that is the only way to safely "consume" a "spurious" NMI and avoid
the 'Dazed' message. Anything else runs the risks of starving the other
subsystems.

> perf NMI comes in, issues pmu_stop 'cleanly' (meaning no spurious
> interrupt).  The cpuc->running bit is never cleared.
>
> kgdb NMI comes in, but the die_chain dictates perf looks at it first.
> perf will see that cpuc->active == 0 and cpuc->running == 1 and bump
> handled.  Thus returning NOTIFY_STOP.  kgdb never sees the NMI. :-(
>
> Now I sent a patch last week that can prevent that extra NMI from being
> generated at the cost of another rdmsrl in the non-pmu_stop cases (which I
> will attach below again, obviously P4 would need something similar too).
>
> I think we currently don't see the problems Stephane describes because the
> only thing we test that uses NMIs are perf, which also happens to be a low
> priority on the die_chain.
>
> But it is an interesting scenario that we should look at more.
>
> Cheers,
> Don
>
>
> diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
> index 48c6d8d..1642f48 100644
> --- a/arch/x86/kernel/cpu/perf_event.c
> +++ b/arch/x86/kernel/cpu/perf_event.c
> @@ -1175,11 +1175,22 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
>                handled++;
>                data.period     = event->hw.last_period;
>
> -               if (!x86_perf_event_set_period(event))
> -                       continue;
> +               /*
> +                * if period is over, process the overflow
> +                * before reseting the counter, otherwise
> +                * a new overflow could occur before the
> +                * event is stopped
> +                */
> +               if (local64_read(&hwc->period_left) <= 0) {
> +                       if (perf_event_overflow(event, 1, &data, regs)) {
> +                               x86_pmu_stop(event, 0);
> +                               continue;
> +                       }
> +                       /* if the overflow doesn't stop the event, resync */
> +                       x86_perf_event_update(event);
> +               }
>
> -               if (perf_event_overflow(event, 1, &data, regs))
> -                       x86_pmu_stop(event, 0);
> +               x86_perf_event_set_period(event);
>        }
>
>        if (handled)
>

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Don Zickus]

On Wed, Sep 29, 2010 at 09:42:26PM +0200, Stephane Eranian wrote:
> On Wed, Sep 29, 2010 at 8:12 PM, Don Zickus <dzickus@redhat.com> wrote:
> > Robert,
> >
> > I think you missed Stephane's point.  Say for example, kgdb is being used
> > while we are doing stuff with the perf counter (and say kgdb's handler is
> > a lower priority than perf; which isn't true I know, but let's say):
> >
> Yes, exactly my point. The reality is you cannot afford to have false positive
> because you may starve another subsystem from an important notification.
> 
> I think it boils down to whether or not we need an error message (Dazed) in
> case no subsystem claimed the NMI. If you were to just silently consume the
> NMI when no subsystem claims it, then you would not have these issues.
> 
> What Don has done is use a heuristic which gets activated when a PMU
> interrupt handler signals that more than one counter have overflowed. His
> claim is that this situation is likely to trigger back-to-back.

Actually its Robert's heuristic. :-)

> 
> The reason this heuristic works is because it waits until ALL the subsystems
> have seen the notification before it declares that the NMI was PMU spurious.
> To do that is uses the DIE_NMI_UNKNOWN callchain. Handler on this chain
> get call last, after all subsystems have seen the notification once. I believe
> that is the only way to safely "consume" a "spurious" NMI and avoid
> the 'Dazed' message. Anything else runs the risks of starving the other
> subsystems.

I agree.

Cheers,
Don

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Robert Richter]

On 29.09.10 15:42:26, Stephane Eranian wrote:
> On Wed, Sep 29, 2010 at 8:12 PM, Don Zickus <dzickus@redhat.com> wrote:
> > I think you missed Stephane's point.  Say for example, kgdb is being used
> > while we are doing stuff with the perf counter (and say kgdb's handler is
> > a lower priority than perf; which isn't true I know, but let's say):
> >
> Yes, exactly my point. The reality is you cannot afford to have false positive
> because you may starve another subsystem from an important notification.

As soon as you stop executing the chain, there are chances to miss an
nmi for other parts of the system. Where is no way to avoid this. So
your argument above is valid also for regular perf nmis and not only
for catched-spurious or back-to-back nmis.

> > Now I sent a patch last week that can prevent that extra NMI from being
> > generated at the cost of another rdmsrl in the non-pmu_stop cases (which I
> > will attach below again, obviously P4 would need something similar too).

A rdmsrl() does not help, it only causes overhead. There is no bit to
detect if a counter overflowed and triggered the interrupt, you only
know the counter value is greater zero or not.

We should take care the discussion becomes not academical and do not
start to overengineer something. I always can imagine some really rare
corner cases in which we may loss an nmi. This is because hardware is
not built for it. But in 99% or so of the cases we get all nmis,
instead of before where all nmis were eaten by the profiler.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Don Zickus]

On Thu, Sep 30, 2010 at 11:12:46AM +0200, Robert Richter wrote:
> On 29.09.10 15:42:26, Stephane Eranian wrote:
> > On Wed, Sep 29, 2010 at 8:12 PM, Don Zickus <dzickus@redhat.com> wrote:
> > > I think you missed Stephane's point.  Say for example, kgdb is being used
> > > while we are doing stuff with the perf counter (and say kgdb's handler is
> > > a lower priority than perf; which isn't true I know, but let's say):
> > >
> > Yes, exactly my point. The reality is you cannot afford to have false positive
> > because you may starve another subsystem from an important notification.
> 
> As soon as you stop executing the chain, there are chances to miss an
> nmi for other parts of the system. Where is no way to avoid this. So
> your argument above is valid also for regular perf nmis and not only
> for catched-spurious or back-to-back nmis.

I don't agree with that.  Most nmi handlers can do a check to see if their
subsystem triggered an nmi or not.  Now we may not catch it in the right
order because one handler is higher in the chain than the other, but
ultimately the other handler will get its chance to execute because it
fired its own nmi (which hasn't been lost).

Whereas the problem Stephane is describing is that the heurestics of the
perf counters 'eats' an NMI, thus possibly starving another handler.  With
back-to-back nmis we are at least polite, letting everyone have a chance to
process the nmi before we indulge ourselves and 'eat' it (if it still
around to be eaten).

However in the case of the 'catched-spurious', we selfishly 'eat' the NMI
without really knowing if it was our to be eaten.  That was the
difference and the concern.

> 
> > > Now I sent a patch last week that can prevent that extra NMI from being
> > > generated at the cost of another rdmsrl in the non-pmu_stop cases (which I
> > > will attach below again, obviously P4 would need something similar too).
> 
> A rdmsrl() does not help, it only causes overhead. There is no bit to
> detect if a counter overflowed and triggered the interrupt, you only
> know the counter value is greater zero or not.

Well, the counters are programmed to trigger an NMI when it crosses zero.
So if we delay reprogramming the counters until after we know if we are
going to issue a pmu_stop, then it should be impossible to trigger an
overflow (because the counters are going to keeping counting above zero,
unless it wraps which would be a different problem all together).

> 
> We should take care the discussion becomes not academical and do not
> start to overengineer something. I always can imagine some really rare
> corner cases in which we may loss an nmi. This is because hardware is
> not built for it. But in 99% or so of the cases we get all nmis,
> instead of before where all nmis were eaten by the profiler.

I don't think this is over engineering.  Basically we haven't seen the
problem yet because the only really active nmi handler is the perf one and
it is designed to be last on the list.  If we start fiddling with
priorities and re-arranging the list, the problem might be exposed quicker
than you think.

Trying to prevent a 'spurious' NMI in the back-to-back case might be a
case for over-engineering, I'll agree to that (I think I tried and
realized how foolish that was).

Cheers,
Don

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Robert Richter]

On 30.09.10 15:44:51, Don Zickus wrote:
> On Thu, Sep 30, 2010 at 11:12:46AM +0200, Robert Richter wrote:

> > As soon as you stop executing the chain, there are chances to miss an
> > nmi for other parts of the system. Where is no way to avoid this. So
> > your argument above is valid also for regular perf nmis and not only
> > for catched-spurious or back-to-back nmis.
> 
> I don't agree with that.  Most nmi handlers can do a check to see if their
> subsystem triggered an nmi or not.  Now we may not catch it in the right
> order because one handler is higher in the chain than the other, but
> ultimately the other handler will get its chance to execute because it
> fired its own nmi (which hasn't been lost).

No, as soon as a handler with higher priority detected an nmi by its
own and handled it, it returns with a stop and all subsequent handlers
get ignored without the chance to check their hardware. So, if perf
consumes an nmi because a counter triggered, there are rare cases that
other handlers may not be executed.

> Whereas the problem Stephane is describing is that the heurestics of the
> perf counters 'eats' an NMI, thus possibly starving another handler.  With
> back-to-back nmis we are at least polite, letting everyone have a chance to
> process the nmi before we indulge ourselves and 'eat' it (if it still
> around to be eaten).
> 
> However in the case of the 'catched-spurious', we selfishly 'eat' the NMI
> without really knowing if it was our to be eaten.  That was the
> difference and the concern.

But, this argument is valid. It would be better to handle
catched-spurious in the 'unknown' path to give other handlers the
chance to check their hardware.

I don't think this is a show-stopper for v2.6.36 even because the perf
handler runs with the lowest priority now. So we will have enough time
after the merge window to improve the code here.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Stephane Eranian]

[repost due to mime-encoding error on my part]

On Fri, Oct 1, 2010 at 9:17 AM, Robert Richter <robert.richter@amd.com>wrote:

> On 30.09.10 15:44:51, Don Zickus wrote:
> > On Thu, Sep 30, 2010 at 11:12:46AM +0200, Robert Richter wrote:
>
> > > As soon as you stop executing the chain, there are chances to miss an
> > > nmi for other parts of the system. Where is no way to avoid this. So
> > > your argument above is valid also for regular perf nmis and not only
> > > for catched-spurious or back-to-back nmis.
> >
> > I don't agree with that.  Most nmi handlers can do a check to see if
> their
> > subsystem triggered an nmi or not.  Now we may not catch it in the right
> > order because one handler is higher in the chain than the other, but
> > ultimately the other handler will get its chance to execute because it
> > fired its own nmi (which hasn't been lost).
>
> No, as soon as a handler with higher priority detected an nmi by its
> own and handled it, it returns with a stop and all subsequent handlers
> get ignored without the chance to check their hardware. So, if perf
> consumes an nmi because a counter triggered, there are rare cases that
> other handlers may not be executed.
>

And that's what you want to avoid.

> > Whereas the problem Stephane is describing is that the heurestics of the
> > perf counters 'eats' an NMI, thus possibly starving another handler.
> > With back-to-back nmis we are at least polite, letting everyone have a chance
> > to process the nmi before we indulge ourselves and 'eat' it (if it still
> > around to be eaten).
> >
> > However in the case of the 'catched-spurious', we selfishly 'eat' the NMI
> > without really knowing if it was our to be eaten.  That was the
> > difference and the concern.
>
> But, this argument is valid. It would be better to handle
> catched-spurious in the 'unknown' path to give other handlers the
> chance to check their hardware.
>

Yes, that was my point. You cannot do it until the kernel invokes the
handler on the 'unknown nmi' callchain.


> I don't think this is a show-stopper for v2.6.36 even because the perf
> handler runs with the lowest priority now. So we will have enough time
> after the merge window to improve the code here.
>
>
That's another issue I have with this NMI callchain logic. It is hard to tell
who's in front of who in each callchain. You may have two registered users
at the same priority, the one which registers last gets priority.

We may not want perf_event to run at the lowest priority because it is
performance sensitive, remember that the counters are running until
you get to the handler. Unlike many of the other subsystems on the
call chain perf_event is doing performance monitoring not debugging.
The rate of calls on the chain is now very high.

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Robert Richter]

On 01.10.10 07:53:37, Stephane Eranian wrote:

> That's another issue I have with this NMI callchain logic. It is hard to tell
> who's in front of who in each callchain. You may have two registered users
> at the same priority, the one which registers last gets priority.
> 
> We may not want perf_event to run at the lowest priority because it is
> performance sensitive, remember that the counters are running until
> you get to the handler. Unlike many of the other subsystems on the
> call chain perf_event is doing performance monitoring not debugging.
> The rate of calls on the chain is now very high.

Yes, actually the perf handler should run with the highest priority to
reduce overhead when executing the handler chain. As this will cause
implications to other handlers I think the most promising approach
will be Andi's suggestion to separate the nmi handlers from the die
chain by adding a new one. We should consider this when reworking the
die handler (cc'ing Huang).

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Stephane Eranian]

On Sat, Oct 2, 2010 at 11:35 AM, Robert Richter <robert.richter@amd.com> wrote:
> On 01.10.10 07:53:37, Stephane Eranian wrote:
>
>> That's another issue I have with this NMI callchain logic. It is hard to tell
>> who's in front of who in each callchain. You may have two registered users
>> at the same priority, the one which registers last gets priority.
>>
>> We may not want perf_event to run at the lowest priority because it is
>> performance sensitive, remember that the counters are running until
>> you get to the handler. Unlike many of the other subsystems on the
>> call chain perf_event is doing performance monitoring not debugging.
>> The rate of calls on the chain is now very high.
>
> Yes, actually the perf handler should run with the highest priority to
> reduce overhead when executing the handler chain. As this will cause
> implications to other handlers I think the most promising approach
> will be Andi's suggestion to separate the nmi handlers from the die
> chain by adding a new one. We should consider this when reworking the
> die handler (cc'ing Huang).
>
I haven't seen Andi's suggestion. But I am guessing he is suggesting
adding a new chain that would be called first and where there would
ONLY be the perf_event subsystem. To handle the spurious PMU
interrupts perf_events would also be added to the lowest priority
list (the one called last) and it would be placed as the last item on
that list (equiv to NMI_UNKNOWN).

Is that the idea?

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Andi Kleen]

> I haven't seen Andi's suggestion. But I am guessing he is suggesting
> adding a new chain that would be called first and where there would
> ONLY be the perf_event subsystem. To handle the spurious PMU

I was thinking of a separate NMI chain 

(and longer term really splitting the die chains into separate chains,
that would solve other problems e.g. with page fault performance
too) 

Possibly that NMI chain could be split up too, but not sure
this is really needed.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Stephane Eranian]

Andi,

On Mon, Oct 4, 2010 at 11:07 AM, Andi Kleen <andi@firstfloor.org> wrote:
>> I haven't seen Andi's suggestion. But I am guessing he is suggesting
>> adding a new chain that would be called first and where there would
>> ONLY be the perf_event subsystem. To handle the spurious PMU
>
> I was thinking of a separate NMI chain
>

You mean separate callbacks which rely on getting an NMI interrupt
from those which don't? If so, I definitively agree with you on that.


> (and longer term really splitting the die chains into separate chains,
> that would solve other problems e.g. with page fault performance
> too)
>
> Possibly that NMI chain could be split up too, but not sure
> this is really needed.
>
You need to split to enforce more explicit priority scheme than what
you  have today with an ordered list per chain.

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Robert Richter]

On 29.09.10 11:33:07, Stephane Eranian wrote:
> There is something else bothering me with cpuc->running.
> 
> It is not reset outside of the interrupt handler. So what if
> event scheduling shuffles things around and an event is
> moved somewhere else. Don't you need to clear the
> cpuc->running[idx] for the old counter index?

We can clear it only in the NMI handler, because the next NMI after
stopping a counter must mark the counter as handled. This may cause
some false positives, but it is the best we can get.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Robert Richter]

On 29.09.10 11:27:45, Cyrill Gorcunov wrote:
> On Wed, Sep 29, 2010 at 05:12:53PM +0200, Robert Richter wrote:
> > On 29.09.10 17:01:40, Robert Richter wrote:
> > > Right, I wasn't remembering correctly, it was P6 and core. And yes, P4
> > > requires the fix. Will send a patch for this.
> > 
> > Ok, Cyrill is sending the patch.
> > 
> > Thanks,
> > 
> > -Robert
> >
> 
> Managed to implement it earlier then I thought.
> 
>   Cyrill
> ---
> Subject: [PATCH -tip] perf, x86: Handle in flight IRQs on P4 platform
> 
> Stephane reported we've missed to guard P4 platform
> against spurious in-flight performance IRQs. Fix it.
> 
> [ the patch is a complement to commit 63e6be6d98e1 ]
> 
> Reported-by: Stephane Eranian <eranian@google.com>
> Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
> CC: Robert Richter <robert.richter@amd.com>
> CC: Lin Ming <ming.m.lin@intel.com>
> ---
>  arch/x86/kernel/cpu/perf_event_p4.c |    6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)

The fix looks correct to me.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Don Zickus]

On Wed, Sep 29, 2010 at 07:27:45PM +0400, Cyrill Gorcunov wrote:
> 
>   Cyrill
> ---
> Subject: [PATCH -tip] perf, x86: Handle in flight IRQs on P4 platform
> 
> Stephane reported we've missed to guard P4 platform
> against spurious in-flight performance IRQs. Fix it.
> 
> [ the patch is a complement to commit 63e6be6d98e1 ]
> 
> Reported-by: Stephane Eranian <eranian@google.com>
> Signed-off-by: Cyrill Gorcunov <gorcunov@openvz.org>
> CC: Robert Richter <robert.richter@amd.com>
> CC: Lin Ming <ming.m.lin@intel.com>

Thanks Cyrill,

I'll test it quickly and try to send it to Ingo.

Cheers,
Don
> ---
>  arch/x86/kernel/cpu/perf_event_p4.c |    6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> Index: linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
> =====================================================================
> --- linux-2.6.git.orig/arch/x86/kernel/cpu/perf_event_p4.c
> +++ linux-2.6.git/arch/x86/kernel/cpu/perf_event_p4.c
> @@ -904,8 +904,12 @@ static int p4_pmu_handle_irq(struct pt_r
>  	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
>  		int overflow;
>  
> -		if (!test_bit(idx, cpuc->active_mask))
> +		if (!test_bit(idx, cpuc->active_mask)) {
> +			/* catch in-flight IRQs */
> +			if (__test_and_clear_bit(idx, cpuc->running))
> +				handled++;
>  			continue;
> +		}
>  
>  		event = cpuc->events[idx];
>  		hwc = &event->hw;

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Robert Richter]

On 29.09.10 09:13:30, Stephane Eranian wrote:

> for_each_set_bit(bit, (unsigned long *)&status, X86_PMC_IDX_MAX) {
>                 struct perf_event *event = cpuc->events[bit];
> 
>                 handled++;
> 
>                 if (!test_bit(bit, cpuc->active_mask))

			/* spurious interrupt here */

>                     continue;
> }
> 
> I think the logic is similar. What makes the difference, it seems, is that
> handled is incremented unconditionally if the ovfl_mask says it has
> an overflow, i.e., before active_mask is checked.

Note that we can use here for_each_set_bit() since we have the status
mask. So we may increment handled always.

On AMD we use for_each_counter(), but only check active counters to
avoid unnecessary rdmsrl()s for unused counters. But here, we only can
increment handled if we detect an overflow or if we know a counter was
disabled.

> On Westmere, we've seen situations where the overflow mask and active
> mask did not agree.

It's the 'spurious interrupt' branch above.

> On counter disable, the overflow mask bit is not cleared, thus one may iterate
> in the loop and fail the active_mask. But handled would be incremented in that
> case, so that would behave like in your patch.

Right, spurious interrupts are counted and a 'handled' is returned.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Stephane Eranian]

On Wed, Sep 29, 2010 at 3:39 PM, Robert Richter <robert.richter@amd.com> wrote:
> On 29.09.10 09:13:30, Stephane Eranian wrote:
>
>> for_each_set_bit(bit, (unsigned long *)&status, X86_PMC_IDX_MAX) {
>>                 struct perf_event *event = cpuc->events[bit];
>>
>>                 handled++;
>>
>>                 if (!test_bit(bit, cpuc->active_mask))
>
>                        /* spurious interrupt here */
>
>>                     continue;
>> }
>>
>> I think the logic is similar. What makes the difference, it seems, is that
>> handled is incremented unconditionally if the ovfl_mask says it has
>> an overflow, i.e., before active_mask is checked.
>
> Note that we can use here for_each_set_bit() since we have the status
> mask. So we may increment handled always.
>
> On AMD we use for_each_counter(), but only check active counters to
> avoid unnecessary rdmsrl()s for unused counters. But here, we only can
> increment handled if we detect an overflow or if we know a counter was
> disabled.
>
>> On Westmere, we've seen situations where the overflow mask and active
>> mask did not agree.
>
> It's the 'spurious interrupt' branch above.
>
>> On counter disable, the overflow mask bit is not cleared, thus one may iterate
>> in the loop and fail the active_mask. But handled would be incremented in that
>> case, so that would behave like in your patch.
>
> Right, spurious interrupts are counted and a 'handled' is returned.
>
Ok, I think we agree on this. It is handled in the Intel case, though it
is not clearly explained with a comment.
The P4 case needs to be fixed.

Here is another difference I noticed in x86_handle_irq() vs.
intel_pmu_handle_irq().
For Intel, handled is incremented even if there is no 64-bit overflow.

With generic X86, it is incremented only when you have a 64-bit
overflow. I think that's wrong. You don't hit that condition very often
on AMD because counters are 47 bits wide, but this is generic code
and on P6 you definitively will. I believe you need to hoist handled++
just after the check on active_mask.


What do you think?

> -Robert
>
> --
> Advanced Micro Devices, Inc.
> Operating System Research Center
>
>

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Stephane Eranian]

On Wed, Sep 29, 2010 at 3:56 PM, Stephane Eranian <eranian@google.com> wrote:
> On Wed, Sep 29, 2010 at 3:39 PM, Robert Richter <robert.richter@amd.com> wrote:
>> On 29.09.10 09:13:30, Stephane Eranian wrote:
>>
>>> for_each_set_bit(bit, (unsigned long *)&status, X86_PMC_IDX_MAX) {
>>>                 struct perf_event *event = cpuc->events[bit];
>>>
>>>                 handled++;
>>>
>>>                 if (!test_bit(bit, cpuc->active_mask))
>>
>>                        /* spurious interrupt here */
>>
>>>                     continue;
>>> }
>>>
>>> I think the logic is similar. What makes the difference, it seems, is that
>>> handled is incremented unconditionally if the ovfl_mask says it has
>>> an overflow, i.e., before active_mask is checked.
>>
>> Note that we can use here for_each_set_bit() since we have the status
>> mask. So we may increment handled always.
>>
>> On AMD we use for_each_counter(), but only check active counters to
>> avoid unnecessary rdmsrl()s for unused counters. But here, we only can
>> increment handled if we detect an overflow or if we know a counter was
>> disabled.
>>
>>> On Westmere, we've seen situations where the overflow mask and active
>>> mask did not agree.
>>
>> It's the 'spurious interrupt' branch above.
>>
>>> On counter disable, the overflow mask bit is not cleared, thus one may iterate
>>> in the loop and fail the active_mask. But handled would be incremented in that
>>> case, so that would behave like in your patch.
>>
>> Right, spurious interrupts are counted and a 'handled' is returned.
>>
> Ok, I think we agree on this. It is handled in the Intel case, though it
> is not clearly explained with a comment.
> The P4 case needs to be fixed.
>
> Here is another difference I noticed in x86_handle_irq() vs.
> intel_pmu_handle_irq().
> For Intel, handled is incremented even if there is no 64-bit overflow.
>
> With generic X86, it is incremented only when you have a 64-bit
> overflow. I think that's wrong. You don't hit that condition very often
> on AMD because counters are 47 bits wide, but this is generic code
> and on P6 you definitively will. I believe you need to hoist handled++
> just after the check on active_mask.
>
>
> What do you think?
>
In other words, I think handled is there to track interrupts, i.e., hw
counter overflows, and not 64-bit software counter overflows (which
do trigger sample recording).

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Robert Richter]

On 29.09.10 10:00:54, Stephane Eranian wrote:

> > Here is another difference I noticed in x86_handle_irq() vs.
> > intel_pmu_handle_irq().
> > For Intel, handled is incremented even if there is no 64-bit overflow.
> >
> > With generic X86, it is incremented only when you have a 64-bit
> > overflow. I think that's wrong. You don't hit that condition very often
> > on AMD because counters are 47 bits wide, but this is generic code
> > and on P6 you definitively will. I believe you need to hoist handled++
> > just after the check on active_mask.
> >
> >
> > What do you think?
> >
> In other words, I think handled is there to track interrupts, i.e., hw
> counter overflows, and not 64-bit software counter overflows (which
> do trigger sample recording).

Stephane, the code looks good. We must first read the counter msr, its
raw value is returned by x86_perf_event_update(). Then we check we MSB
of the *counter* value and if it is zero, we detected a counter
overflow (not a 64 bit overflow) and increment 'handled'.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Stephane Eranian]

On Sat, Oct 2, 2010 at 11:50 AM, Robert Richter <robert.richter@amd.com> wrote:
> On 29.09.10 10:00:54, Stephane Eranian wrote:
>
>> > Here is another difference I noticed in x86_handle_irq() vs.
>> > intel_pmu_handle_irq().
>> > For Intel, handled is incremented even if there is no 64-bit overflow.
>> >
>> > With generic X86, it is incremented only when you have a 64-bit
>> > overflow. I think that's wrong. You don't hit that condition very often
>> > on AMD because counters are 47 bits wide, but this is generic code
>> > and on P6 you definitively will. I believe you need to hoist handled++
>> > just after the check on active_mask.
>> >
>> >
>> > What do you think?
>> >
>> In other words, I think handled is there to track interrupts, i.e., hw
>> counter overflows, and not 64-bit software counter overflows (which
>> do trigger sample recording).
>
> Stephane, the code looks good. We must first read the counter msr, its
> raw value is returned by x86_perf_event_update(). Then we check we MSB
> of the *counter* value and if it is zero, we detected a counter
> overflow (not a 64 bit overflow) and increment 'handled'.
>
Yes, you are right. I double-checked that again Friday and the code is
okay for Intel, generic X86 and P4.

Re: [tip:perf/urgent] perf, x86: Catch spurious interrupts after disabling counters

[Cyrill Gorcunov]

On Wed, Sep 29, 2010 at 03:56:08PM +0200, Stephane Eranian wrote:
...
> Ok, I think we agree on this. It is handled in the Intel case, though it
> is not clearly explained with a comment.
> The P4 case needs to be fixed.
>

Seems so. Thanks Stephane. I'll check and fix as only be able to
(in a couple of hours).

  Cyrill

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Peter Zijlstra]

On Fri, 2010-09-10 at 17:56 +0200, Robert Richter wrote:
> The commit:
> 
>  e40b172 x86: Move notify_die from nmi.c to traps.c
> 
> moved the nmi handler call to default_do_nmi(). DIE_NMI_IPI and
> DIE_NMI are called subsequently now. If the return code is
> !NOTIFY_STOP, then the handlers are called twice. This patch fixes
> this. 

What is this NMI_IPI thing, and why do we bother?

If the hardware reason thin can discriminate between IPI and other NMIs
we should never call both chains, if it cannot, why do we have 2 chains?

In any case, I don't think perf/oprofile/etc,. should use NMI_IPI, that
doesn't really make sense.

Or am I totally confused about things here?

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Stephane Eranian]

On Thu, Sep 16, 2010 at 7:42 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Fri, 2010-09-10 at 17:56 +0200, Robert Richter wrote:
>> The commit:
>>
>>  e40b172 x86: Move notify_die from nmi.c to traps.c
>>
>> moved the nmi handler call to default_do_nmi(). DIE_NMI_IPI and
>> DIE_NMI are called subsequently now. If the return code is
>> !NOTIFY_STOP, then the handlers are called twice. This patch fixes
>> this.
>
> What is this NMI_IPI thing, and why do we bother?
>
I think you only need to be on one of the callchains and most likely
the higher priority one. That's beauce you're using NMI for performance
monitoring and not debugging. Thus you expect a high rate of calls and
you don't want to be calling a bunch of other handlers for nothing.

The way the code is laid out in default_do_nmi() means that NMI_IPI
has higher priority over NMI. NMI also depends on X86_LOCAL_APIC,
but without it no PMU interrupts anyway.

The other thing to notice is that default_do_nmi() is the only place for
invoking NMI_IPI. I don't see the point of it vs. NMI (except APIC vs. no APIC).

Why not just have one level, NMI, and use the priority field to position
handlers correctly?

> If the hardware reason thin can discriminate between IPI and other NMIs
> we should never call both chains, if it cannot, why do we have 2 chains?
>
> In any case, I don't think perf/oprofile/etc,. should use NMI_IPI, that
> doesn't really make sense.
>
> Or am I totally confused about things here?
>

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Peter Zijlstra]

On Thu, 2010-09-16 at 22:18 +0200, Stephane Eranian wrote:
> 
> Why not just have one level, NMI, and use the priority field to position
> handlers correctly? 

My point exactly.. I just don't know the history of this code, I had a
quick peek at git but it seems to predate .12, I'll try and dig through
the hysterical tree later today.

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Huang Ying]

On Fri, 2010-09-17 at 01:42 +0800, Peter Zijlstra wrote:
> On Fri, 2010-09-10 at 17:56 +0200, Robert Richter wrote:
> > The commit:
> > 
> >  e40b172 x86: Move notify_die from nmi.c to traps.c
> > 
> > moved the nmi handler call to default_do_nmi(). DIE_NMI_IPI and
> > DIE_NMI are called subsequently now. If the return code is
> > !NOTIFY_STOP, then the handlers are called twice. This patch fixes
> > this. 
> 
> What is this NMI_IPI thing, and why do we bother?
> 
> If the hardware reason thin can discriminate between IPI and other NMIs
> we should never call both chains, if it cannot, why do we have 2 chains?
> 
> In any case, I don't think perf/oprofile/etc,. should use NMI_IPI, that
> doesn't really make sense.
> 
> Or am I totally confused about things here?

Please take a look at my recent patch that rewrites the NMI handler.
Where NMI are divided into two types. CPU specific NMI are processed
firstly as DIE_NMI_IPI, and non-CPU specific NMI are processed as
DIE_NMI.

CPU specific NMIs include perf event, oprofile, crash, etc. They can be
processed on specific CPU only.

non-CPU specific NMIs include SERR and IOCHK NMI (via checking port
0x61) and APEI GHES NMI. They can be processed on any CPU.

Best Regards,
Huang Ying

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Peter Zijlstra]

On Fri, 2010-09-17 at 08:13 +0800, Huang Ying wrote:

> Please take a look at my recent patch that rewrites the NMI handler.

(Link would have been useful)

> Where NMI are divided into two types. CPU specific NMI are processed
> firstly as DIE_NMI_IPI, and non-CPU specific NMI are processed as
> DIE_NMI.

OK, and you can discriminate between these two by means of that reason
port? Still I think NMI_IPI is a terrible name for that.

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Robert Richter]

On 17.09.10 03:52:16, Peter Zijlstra wrote:
> On Fri, 2010-09-17 at 08:13 +0800, Huang Ying wrote:
> 
> > Please take a look at my recent patch that rewrites the NMI handler.
> 
> (Link would have been useful)
> 
> > Where NMI are divided into two types. CPU specific NMI are processed
> > firstly as DIE_NMI_IPI, and non-CPU specific NMI are processed as
> > DIE_NMI.
> 
> OK, and you can discriminate between these two by means of that reason
> port? Still I think NMI_IPI is a terrible name for that.

I think the current implementation that devides into NMI and NMI_IPI
is just to keep a certain order when calling the handlers - handling
local NMIs before non-CPU NMIs.

The patch I send is wrong, it should actually remove DIE_NMI_IPI. But
in the end the result is the same, not to call the perf handler twice.

I had this in my patch queue and posted it for debugging the 'unknown
nmi' warnings. Huang also did the same change in his patch set, but
his patch included too much changes and was not usable as a single
fix.

I am fine with dropping this patch in favor of Huang's patches, but
maybe we extract a single patch of this change for stable too.

-Robert

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Cyrill Gorcunov]

On Fri, Sep 17, 2010 at 10:13:58AM +0200, Robert Richter wrote:
> On 17.09.10 03:52:16, Peter Zijlstra wrote:
> > On Fri, 2010-09-17 at 08:13 +0800, Huang Ying wrote:
> > 
> > > Please take a look at my recent patch that rewrites the NMI handler.
> > 
> > (Link would have been useful)
> > 
> > > Where NMI are divided into two types. CPU specific NMI are processed
> > > firstly as DIE_NMI_IPI, and non-CPU specific NMI are processed as
> > > DIE_NMI.
> > 
> > OK, and you can discriminate between these two by means of that reason
> > port? Still I think NMI_IPI is a terrible name for that.
> 
> I think the current implementation that devides into NMI and NMI_IPI
> is just to keep a certain order when calling the handlers - handling
> local NMIs before non-CPU NMIs.
> 
...

This DIE_NMI_IPI definitely strange. Perhaps we could rename it to something
more understandable and implement separate notifier chain for it so that
code would not need to check if (... != DIE_NMI_IPI). Hm?

	-- Cyrill

Re: [PATCH] x86: fix duplicate calls of the nmi handler

[Huang Ying]

On Fri, 2010-09-17 at 15:52 +0800, Peter Zijlstra wrote:
> On Fri, 2010-09-17 at 08:13 +0800, Huang Ying wrote:
> 
> > Please take a look at my recent patch that rewrites the NMI handler.
> 
> (Link would have been useful)

http://lkml.org/lkml/2010/9/9/575

> > Where NMI are divided into two types. CPU specific NMI are processed
> > firstly as DIE_NMI_IPI, and non-CPU specific NMI are processed as
> > DIE_NMI.
> 
> OK, and you can discriminate between these two by means of that reason
> port? Still I think NMI_IPI is a terrible name for that.

In my patches, DIE_NMI_IPI go firstly, then DIE_NMI follows. NMI reason
is not used for that.

I don't know the history of naming too. But I think the name may come
from apic->send_IPI(mask, NMI_VECTOR). Which is used by crash_kexec,

crash_kexec
  machine_crash_shutdown
    kdump_nmi_shootdown_cpus
      nmi_shootdown_cpus
        smp_send_nmi_allbutself

Best Regards,
Huang Ying

Re: [PATCH 0/3 v2] nmi perf fixes

[Peter Zijlstra]

On Fri, 2010-09-10 at 13:41 +0200, Peter Zijlstra wrote:
> On Thu, 2010-09-02 at 15:07 -0400, Don Zickus wrote:
> > Fixes to allow unknown nmis to pass through the perf nmi handler instead
> > of being swallowed.  Contains patches that are already in Ingo's tree.  Added
> > here for completeness.  Based on ingo/tip

> Both Ingo and I are getting Dazed and confused on our AMD machines, it
> started before yesterday (that is, after backing out all my recent
> changes it still gets dazed), so I suspect this set.
> 
> I'll look at getting a trace of the thing, but if any of you has a
> bright idea...


           <...>-2155  [000]    57.298895: perf_event_nmi_handler: NMI-handled(1): 5735 0 0
           <...>-2155  [000]    57.298896: perf_event_nmi_handler: NMI-stop: 5735 0 0
           <...>-2155  [000]    57.298898: perf_event_nmi_handler: NMI: 5736 0 0
           <...>-2155  [000]    57.298898: x86_pmu_handle_irq: OVERFLOW: 0
           <...>-2155  [000]    57.298899: x86_pmu_handle_irq: HANDLED: 1
           <...>-2155  [000]    57.298901: x86_pmu_handle_irq: OVERFLOW: 2
           <...>-2155  [000]    57.298901: x86_pmu_handle_irq: OVERFLOW: 3
           <...>-2155  [000]    57.298902: perf_event_nmi_handler: NMI-handled(1): 5736 0 0
           <...>-2155  [000]    57.298903: perf_event_nmi_handler: NMI-stop: 5736 0 0
           <...>-2155  [000]    57.298905: perf_event_nmi_handler: NMI: 5737 0 0
           <...>-2155  [000]    57.298905: x86_pmu_handle_irq: OVERFLOW: 0
           <...>-2155  [000]    57.298906: x86_pmu_handle_irq: HANDLED: 1
           <...>-2155  [000]    57.298908: x86_pmu_handle_irq: OVERFLOW: 2
           <...>-2155  [000]    57.298908: x86_pmu_handle_irq: OVERFLOW: 3
           <...>-2155  [000]    57.298909: perf_event_nmi_handler: NMI-handled(1): 5737 0 0
           <...>-2155  [000]    57.298909: perf_event_nmi_handler: NMI-stop: 5737 0 0
           <...>-2155  [000]    57.298911: perf_event_nmi_handler: NMI: 5738 0 0
           <...>-2155  [000]    57.298912: x86_pmu_handle_irq: OVERFLOW: 0
           <...>-2155  [000]    57.298913: x86_pmu_handle_irq: HANDLED: 1
           <...>-2155  [000]    57.298915: x86_pmu_handle_irq: OVERFLOW: 2
           <...>-2155  [000]    57.298916: x86_pmu_handle_irq: OVERFLOW: 3
           <...>-2155  [000]    57.298916: perf_event_nmi_handler: NMI-handled(1): 5738 0 0
           <...>-2155  [000]    57.298917: perf_event_nmi_handler: NMI-stop: 5738 0 0
           <...>-2155  [000]    57.298919: perf_event_nmi_handler: NMI: 5739 0 0
           <...>-2155  [000]    57.298920: x86_pmu_handle_irq: OVERFLOW: 0
           <...>-2155  [000]    57.298921: x86_pmu_handle_irq: OVERFLOW: 2
           <...>-2155  [000]    57.298921: x86_pmu_handle_irq: OVERFLOW: 3
           <...>-2155  [000]    57.298922: perf_event_nmi_handler: NMI-handled(0): 5739 0 0
           <...>-2155  [000]    57.298923: perf_event_nmi_handler: NMI: 5739 0 0
           <...>-2155  [000]    57.298924: x86_pmu_handle_irq: OVERFLOW: 0
           <...>-2155  [000]    57.298925: x86_pmu_handle_irq: OVERFLOW: 2
           <...>-2155  [000]    57.298925: x86_pmu_handle_irq: OVERFLOW: 3
           <...>-2155  [000]    57.298926: perf_event_nmi_handler: NMI-handled(0): 5739 0 0
           <...>-2155  [000]    57.298927: perf_event_nmi_handler: NMI: 5739 0 0
           <...>-2155  [000]    57.298928: perf_event_nmi_handler: NMI-fail


Which suggests that 5738 was a good NMI, 5739 is an unhandled NMI, we
see it twice, once through DIE_NMI once through DIE_NMIUNKNOWN.

The problem seems to be that we don't tag it because handled isn't
larger than 1.

Its easy to reproduce on my opteron, simply run: 
  pref record -fg ./hackbench 50
a few times.

I'll try and reset the PMU on init to clear some of those OVERFLOW msgs
(hmm, maybe also read the ctrl word and check EN and INT before doing
the overflow check)..

---
 arch/x86/kernel/cpu/perf_event.c |   34 +++++++++++++++++++++++++++++++++-
 1 files changed, 33 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index de6569c..9aff608 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1127,6 +1127,15 @@ static void x86_pmu_disable(struct perf_event *event)
 	perf_event_update_userpage(event);
 }
 
+static int pmc_overflow(int idx)
+{
+	u64 val;
+
+	rdmsrl(x86_pmu.perfctr + idx, val);
+
+	return !!(val & (1ULL << (x86_pmu.cntval_bits - 1)));
+}
+
 static int x86_pmu_handle_irq(struct pt_regs *regs)
 {
 	struct perf_sample_data data;
@@ -1141,6 +1150,8 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 	cpuc = &__get_cpu_var(cpu_hw_events);
 
 	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
+		if (pmc_overflow(idx))
+			trace_printk("OVERFLOW: %d\n", idx);
 		if (!test_bit(idx, cpuc->active_mask))
 			continue;
 
@@ -1154,6 +1165,7 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 		/*
 		 * event overflow
 		 */
+		trace_printk("HANDLED: %d\n", idx);
 		handled++;
 		data.period	= event->hw.last_period;
 
@@ -1215,6 +1227,11 @@ perf_event_nmi_handler(struct notifier_block *self,
 	unsigned int this_nmi;
 	int handled;
 
+	trace_printk("NMI: %d %d %d\n",
+			percpu_read(irq_stat.__nmi_count),
+			__get_cpu_var(pmu_nmi).marked,
+			__get_cpu_var(pmu_nmi).handled);
+
 	if (!atomic_read(&active_events))
 		return NOTIFY_DONE;
 
@@ -1224,9 +1241,12 @@ perf_event_nmi_handler(struct notifier_block *self,
 		break;
 	case DIE_NMIUNKNOWN:
 		this_nmi = percpu_read(irq_stat.__nmi_count);
-		if (this_nmi != __get_cpu_var(pmu_nmi).marked)
+		if (this_nmi != __get_cpu_var(pmu_nmi).marked) {
+			trace_printk("NMI-fail\n");
 			/* let the kernel handle the unknown nmi */
 			return NOTIFY_DONE;
+		}
+		trace_printk("NMI-consume\n");
 		/*
 		 * This one is a PMU back-to-back nmi. Two events
 		 * trigger 'simultaneously' raising two back-to-back
@@ -1242,6 +1262,13 @@ perf_event_nmi_handler(struct notifier_block *self,
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 
 	handled = x86_pmu.handle_irq(args->regs);
+
+	trace_printk("NMI-handled(%d): %d %d %d\n",
+			handled,
+			percpu_read(irq_stat.__nmi_count),
+			__get_cpu_var(pmu_nmi).marked,
+			__get_cpu_var(pmu_nmi).handled);
+
 	if (!handled)
 		return NOTIFY_DONE;
 
@@ -1264,6 +1291,11 @@ perf_event_nmi_handler(struct notifier_block *self,
 		__get_cpu_var(pmu_nmi).handled	= handled;
 	}
 
+	trace_printk("NMI-stop: %d %d %d\n",
+			percpu_read(irq_stat.__nmi_count),
+			__get_cpu_var(pmu_nmi).marked,
+			__get_cpu_var(pmu_nmi).handled);
+
 	return NOTIFY_STOP;
 }
 

Re: [PATCH 0/3 v2] nmi perf fixes

[Peter Zijlstra]

On Fri, 2010-09-10 at 15:34 +0200, Peter Zijlstra wrote:
> > I'll look at getting a trace of the thing, but if any of you has a
> > bright idea...
> 
           <...>-3164  [000]    51.658621: perf_event_nmi_handler: NMI: 14146 6078 1
           <...>-3164  [000]    51.658622: x86_pmu_handle_irq: OVERFLOW: 1
           <...>-3164  [000]    51.658622: x86_pmu_handle_irq: HANDLED: 1
           <...>-3164  [000]    51.658624: perf_event_nmi_handler: NMI-handled(1): 14146 6078 1
           <...>-3164  [000]    51.658625: perf_event_nmi_handler: NMI-stop: 14146 6078 1
           <...>-3164  [000]    51.658627: perf_event_nmi_handler: NMI: 14147 6078 1
           <...>-3164  [000]    51.658627: x86_pmu_handle_irq: OVERFLOW: 1
           <...>-3164  [000]    51.658628: x86_pmu_handle_irq: HANDLED: 1
           <...>-3164  [000]    51.658631: perf_event_nmi_handler: NMI-handled(1): 14147 6078 1
           <...>-3164  [000]    51.658631: perf_event_nmi_handler: NMI-stop: 14147 6078 1
           <...>-3164  [000]    51.658633: perf_event_nmi_handler: NMI: 14148 6078 1
           <...>-3164  [000]    51.658634: perf_event_nmi_handler: NMI-handled(0): 14148 6078 1
           <...>-3164  [000]    51.658635: perf_event_nmi_handler: NMI: 14148 6078 1
           <...>-3164  [000]    51.658636: perf_event_nmi_handler: NMI-handled(0): 14148 6078 1
           <...>-3164  [000]    51.658637: perf_event_nmi_handler: NMI: 14148 6078 1
           <...>-3164  [000]    51.658637: perf_event_nmi_handler: NMI-fail

That seems to be clear enough.. no idea where that extra NMI comes from.
Robert any clue?

---
 arch/x86/kernel/cpu/perf_event.c |   39 +++++++++++++++++++++++++++++++++++++-
 1 files changed, 38 insertions(+), 1 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index de6569c..5e9921c 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1127,6 +1127,20 @@ static void x86_pmu_disable(struct perf_event *event)
 	perf_event_update_userpage(event);
 }
 
+static int pmc_overflow(int idx)
+{
+	u64 val;
+
+	rdmsrl(x86_pmu.eventsel + idx, val);
+	if ((val & (ARCH_PERFMON_EVENTSEL_ENABLE | ARCH_PERFMON_EVENTSEL_INT)) ==
+		  (ARCH_PERFMON_EVENTSEL_ENABLE | ARCH_PERFMON_EVENTSEL_INT)) {
+		rdmsrl(x86_pmu.perfctr + idx, val);
+		return !(val & (1ULL << (x86_pmu.cntval_bits - 1)));
+	}
+
+	return 0;
+}
+
 static int x86_pmu_handle_irq(struct pt_regs *regs)
 {
 	struct perf_sample_data data;
@@ -1141,6 +1155,8 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 	cpuc = &__get_cpu_var(cpu_hw_events);
 
 	for (idx = 0; idx < x86_pmu.num_counters; idx++) {
+		if (pmc_overflow(idx))
+			trace_printk("OVERFLOW: %d\n", idx);
 		if (!test_bit(idx, cpuc->active_mask))
 			continue;
 
@@ -1154,6 +1170,7 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 		/*
 		 * event overflow
 		 */
+		trace_printk("HANDLED: %d\n", idx);
 		handled++;
 		data.period	= event->hw.last_period;
 
@@ -1215,6 +1232,11 @@ perf_event_nmi_handler(struct notifier_block *self,
 	unsigned int this_nmi;
 	int handled;
 
+	trace_printk("NMI: %d %d %d\n",
+			percpu_read(irq_stat.__nmi_count),
+			__get_cpu_var(pmu_nmi).marked,
+			__get_cpu_var(pmu_nmi).handled);
+
 	if (!atomic_read(&active_events))
 		return NOTIFY_DONE;
 
@@ -1224,9 +1246,12 @@ perf_event_nmi_handler(struct notifier_block *self,
 		break;
 	case DIE_NMIUNKNOWN:
 		this_nmi = percpu_read(irq_stat.__nmi_count);
-		if (this_nmi != __get_cpu_var(pmu_nmi).marked)
+		if (this_nmi != __get_cpu_var(pmu_nmi).marked) {
+			trace_printk("NMI-fail\n");
 			/* let the kernel handle the unknown nmi */
 			return NOTIFY_DONE;
+		}
+		trace_printk("NMI-consume\n");
 		/*
 		 * This one is a PMU back-to-back nmi. Two events
 		 * trigger 'simultaneously' raising two back-to-back
@@ -1242,6 +1267,13 @@ perf_event_nmi_handler(struct notifier_block *self,
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 
 	handled = x86_pmu.handle_irq(args->regs);
+
+	trace_printk("NMI-handled(%d): %d %d %d\n",
+			handled,
+			percpu_read(irq_stat.__nmi_count),
+			__get_cpu_var(pmu_nmi).marked,
+			__get_cpu_var(pmu_nmi).handled);
+
 	if (!handled)
 		return NOTIFY_DONE;
 
@@ -1264,6 +1296,11 @@ perf_event_nmi_handler(struct notifier_block *self,
 		__get_cpu_var(pmu_nmi).handled	= handled;
 	}
 
+	trace_printk("NMI-stop: %d %d %d\n",
+			percpu_read(irq_stat.__nmi_count),
+			__get_cpu_var(pmu_nmi).marked,
+			__get_cpu_var(pmu_nmi).handled);
+
 	return NOTIFY_STOP;
 }
 

Re: [PATCH 0/3 v2] nmi perf fixes

[Cyrill Gorcunov]

On Fri, Sep 10, 2010 at 03:52:50PM +0200, Peter Zijlstra wrote:
... 
> That seems to be clear enough.. no idea where that extra NMI comes from.
> Robert any clue?
> 
...

For what is worth, I've tested amd turion box -- not affected.

	-- Cyrill

Re: [PATCH 0/3 v2] nmi perf fixes

[Stephane Eranian]

Robert,

I have able to reproduce very easily on:

processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 16
model		: 2
model name	: AMD Phenom(tm) 8450 Triple-Core Processor
stepping	: 3

using:

$ perf record -g ./hackbench 10
Time: 0.415
[ perf record: Woken up 3 times to write data ]
[ perf record: Captured and wrote 0.706 MB perf.data (~30841 samples) ]
$ dmesg | tail
[  418.946001] Uhhuh. NMI received for unknown reason 3d on CPU 0.
[  418.946001] Do you have a strange power saving mode enabled?
[  418.946001] Dazed and confused, but trying to continue

This is with 2.6.36-rc4-tip as of this morning from tip-x86.


On Mon, Sep 13, 2010 at 10:55 AM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> On Fri, Sep 10, 2010 at 03:52:50PM +0200, Peter Zijlstra wrote:
> ...
>> That seems to be clear enough.. no idea where that extra NMI comes from.
>> Robert any clue?
>>
> ...
>
> For what is worth, I've tested amd turion box -- not affected.
>
>        -- Cyrill
>

Re: [PATCH 0/3 v2] nmi perf fixes

[Cyrill Gorcunov]

On Mon, Sep 13, 2010 at 11:54:59AM +0200, Stephane Eranian wrote:
> Robert,
> 
> I have able to reproduce very easily on:
> 
> processor	: 0
> vendor_id	: AuthenticAMD
> cpu family	: 16
> model		: 2
> model name	: AMD Phenom(tm) 8450 Triple-Core Processor
> stepping	: 3
> 
> using:
> 
> $ perf record -g ./hackbench 10
> Time: 0.415
> [ perf record: Woken up 3 times to write data ]
> [ perf record: Captured and wrote 0.706 MB perf.data (~30841 samples) ]
> $ dmesg | tail
> [  418.946001] Uhhuh. NMI received for unknown reason 3d on CPU 0.
> [  418.946001] Do you have a strange power saving mode enabled?
> [  418.946001] Dazed and confused, but trying to continue
> 
> This is with 2.6.36-rc4-tip as of this morning from tip-x86.
> 

Hi Stephane, the hackbench is Ingo's one? I've been using
http://people.redhat.com/mingo/cfs-scheduler/tools/hackbench.c

	-- Cyrill

Re: [PATCH 0/3 v2] nmi perf fixes

[Stephane Eranian]

On Mon, Sep 13, 2010 at 12:07 PM, Cyrill Gorcunov <gorcunov@gmail.com> wrote:
> On Mon, Sep 13, 2010 at 11:54:59AM +0200, Stephane Eranian wrote:
>> Robert,
>>
>> I have able to reproduce very easily on:
>>
>> processor     : 0
>> vendor_id     : AuthenticAMD
>> cpu family    : 16
>> model         : 2
>> model name    : AMD Phenom(tm) 8450 Triple-Core Processor
>> stepping      : 3
>>
>> using:
>>
>> $ perf record -g ./hackbench 10
>> Time: 0.415
>> [ perf record: Woken up 3 times to write data ]
>> [ perf record: Captured and wrote 0.706 MB perf.data (~30841 samples) ]
>> $ dmesg | tail
>> [  418.946001] Uhhuh. NMI received for unknown reason 3d on CPU 0.
>> [  418.946001] Do you have a strange power saving mode enabled?
>> [  418.946001] Dazed and confused, but trying to continue
>>
>> This is with 2.6.36-rc4-tip as of this morning from tip-x86.
>>
>
> Hi Stephane, the hackbench is Ingo's one? I've been using
> http://people.redhat.com/mingo/cfs-scheduler/tools/hackbench.c
>
No, it's one I found via google search:

http://devresources.linuxfoundation.org/craiger/hackbench/

Re: [PATCH 0/3 v2] nmi perf fixes

[Cyrill Gorcunov]

On Mon, Sep 13, 2010 at 12:10:00PM +0200, Stephane Eranian wrote:
...
> > Hi Stephane, the hackbench is Ingo's one? I've been using
> > http://people.redhat.com/mingo/cfs-scheduler/tools/hackbench.c
> >
> No, it's one I found via google search:
> 
> http://devresources.linuxfoundation.org/craiger/hackbench/
> 

ok, tried it as well -- no NMI issue.

	-- Cyrill

[PATCH -v3] perf, x86: try to handle unknown nmis with running perfctrs

[Robert Richter]

Peter,

this is version 3 without timestamp code. Compared to -v1 it
implements the handling for 2-1-0 nmi sequences.

-Robert

>From 71a206394d8a3536b033247ec14ad037fef77236 Mon Sep 17 00:00:00 2001
From: Robert Richter <robert.richter@amd.com>
Date: Tue, 17 Aug 2010 16:42:03 +0200
Subject: [PATCH] perf, x86: try to handle unknown nmis with running perfctrs

When perfctrs are running it is valid to have unhandled nmis, two
events could trigger 'simultaneously' raising two back-to-back
NMIs. If the first NMI handles both, the latter will be empty and daze
the CPU.

The solution to avoid an 'unknown nmi' massage in this case was simply
to stop the nmi handler chain when perfctrs are runnning by stating
the nmi was handled. This has the drawback that a) we can not detect
unknown nmis anymore, and b) subsequent nmi handlers are not called.

This patch addresses this. Now, we check this unknown NMI if it could
be a perfctr back-to-back NMI. Otherwise we pass it and let the kernel
handle the unknown nmi.

This is a debug log:

cpu #6, nmi #32333, skip_nmi #32330, handled = 1, time = 1934364430
cpu #6, nmi #32334, skip_nmi #32330, handled = 1, time = 1934704616
cpu #6, nmi #32335, skip_nmi #32336, handled = 2, time = 1936032320
cpu #6, nmi #32336, skip_nmi #32336, handled = 0, time = 1936034139
cpu #6, nmi #32337, skip_nmi #32336, handled = 1, time = 1936120100
cpu #6, nmi #32338, skip_nmi #32336, handled = 1, time = 1936404607
cpu #6, nmi #32339, skip_nmi #32336, handled = 1, time = 1937983416
cpu #6, nmi #32340, skip_nmi #32341, handled = 2, time = 1938201032
cpu #6, nmi #32341, skip_nmi #32341, handled = 0, time = 1938202830
cpu #6, nmi #32342, skip_nmi #32341, handled = 1, time = 1938443743
cpu #6, nmi #32343, skip_nmi #32341, handled = 1, time = 1939956552
cpu #6, nmi #32344, skip_nmi #32341, handled = 1, time = 1940073224
cpu #6, nmi #32345, skip_nmi #32341, handled = 1, time = 1940485677
cpu #6, nmi #32346, skip_nmi #32347, handled = 2, time = 1941947772
cpu #6, nmi #32347, skip_nmi #32347, handled = 1, time = 1941949818
cpu #6, nmi #32348, skip_nmi #32347, handled = 0, time = 1941951591
Uhhuh. NMI received for unknown reason 00 on CPU 6.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue

Deltas:

nmi #32334 340186
nmi #32335 1327704
nmi #32336 1819      <<<< back-to-back nmi [1]
nmi #32337 85961
nmi #32338 284507
nmi #32339 1578809
nmi #32340 217616
nmi #32341 1798      <<<< back-to-back nmi [2]
nmi #32342 240913
nmi #32343 1512809
nmi #32344 116672
nmi #32345 412453
nmi #32346 1462095   <<<< 1st nmi (standard) handling 2 counters
nmi #32347 2046      <<<< 2nd nmi (back-to-back) handling one counter
nmi #32348 1773      <<<< 3rd nmi (back-to-back) handling no counter! [3]

For  back-to-back nmi detection there are the following rules:

The perfctr nmi handler was handling more than one counter and no
counter was handled in the subsequent nmi (see [1] and [2] above).

There is another case if there are two subsequent back-to-back nmis
[3]. The 2nd is detected as back-to-back because the first handled
more than one counter. If the second handles one counter and the 3rd
handles nothing, we drop the 3rd nmi because it could be a
back-to-back nmi.

Signed-off-by: Robert Richter <robert.richter@amd.com>
---
 arch/x86/kernel/cpu/perf_event.c |   65 ++++++++++++++++++++++++++++++-------
 1 files changed, 52 insertions(+), 13 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index f2da20f..bbcc89e 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1154,7 +1154,7 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 		/*
 		 * event overflow
 		 */
-		handled		= 1;
+		handled++;
 		data.period	= event->hw.last_period;
 
 		if (!x86_perf_event_set_period(event))
@@ -1200,12 +1200,20 @@ void perf_events_lapic_init(void)
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 }
 
+struct perfctr_nmi {
+	unsigned int	marked;
+	int		handled;
+};
+
+static DEFINE_PER_CPU(struct perfctr_nmi, nmi);
+
 static int __kprobes
 perf_event_nmi_handler(struct notifier_block *self,
 			 unsigned long cmd, void *__args)
 {
 	struct die_args *args = __args;
-	struct pt_regs *regs;
+	unsigned int this_nmi;
+	int handled;
 
 	if (!atomic_read(&active_events))
 		return NOTIFY_DONE;
@@ -1214,22 +1222,53 @@ perf_event_nmi_handler(struct notifier_block *self,
 	case DIE_NMI:
 	case DIE_NMI_IPI:
 		break;
-
+	case DIE_NMIUNKNOWN:
+		this_nmi = percpu_read(irq_stat.__nmi_count);
+		if (this_nmi != __get_cpu_var(nmi).marked)
+			/* let the kernel handle the unknown nmi */
+			return NOTIFY_DONE;
+		/*
+		 * This one is a perfctr back-to-back nmi. Two events
+		 * trigger 'simultaneously' raising two back-to-back
+		 * NMIs. If the first NMI handles both, the latter
+		 * will be empty and daze the CPU. So, we drop it to
+		 * avoid false-positive 'unknown nmi' messages.
+		 */
+		return NOTIFY_STOP;
 	default:
 		return NOTIFY_DONE;
 	}
 
-	regs = args->regs;
-
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
-	/*
-	 * Can't rely on the handled return value to say it was our NMI, two
-	 * events could trigger 'simultaneously' raising two back-to-back NMIs.
-	 *
-	 * If the first NMI handles both, the latter will be empty and daze
-	 * the CPU.
-	 */
-	x86_pmu.handle_irq(regs);
+
+	handled = x86_pmu.handle_irq(args->regs);
+	if (!handled)
+		return NOTIFY_DONE;
+
+	this_nmi = percpu_read(irq_stat.__nmi_count);
+	if (handled > 1)
+		goto mark_nmi;
+	if ((__get_cpu_var(nmi).marked == this_nmi)
+	    && (__get_cpu_var(nmi).handled > 1))
+		/*
+		 * We could have two subsequent back-to-back nmis: The
+		 * first handles more than one counter, the 2nd
+		 * handles only one counter and the 3rd handles no
+		 * counter.
+		 *
+		 * This is the 2nd nmi because the previous was
+		 * handling more than one counter. We will mark the
+		 * next (3rd) and then drop it if unhandled.
+		 */
+		goto mark_nmi;
+
+	/* this may not trigger back-to-back nmis */
+	return NOTIFY_STOP;
+
+ mark_nmi:
+	/* the next nmi could be a back-to-back nmi */
+	__get_cpu_var(nmi).marked	= this_nmi + 1;
+	__get_cpu_var(nmi).handled	= handled;
 
 	return NOTIFY_STOP;
 }
-- 
1.7.1.1

-- 
Advanced Micro Devices, Inc.
Operating System Research Center

[tip:perf/urgent] perf, x86: Try to handle unknown nmis with an enabled PMU

[tip-bot for Robert Richter]

Commit-ID:  8e3e42b4f88602bb6e1f0b74afd80ff4703f79ce
Gitweb:     http://git.kernel.org/tip/8e3e42b4f88602bb6e1f0b74afd80ff4703f79ce
Author:     Robert Richter <robert.richter@amd.com>
AuthorDate: Tue, 17 Aug 2010 16:42:03 +0200
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 20 Aug 2010 15:00:05 +0200

perf, x86: Try to handle unknown nmis with an enabled PMU

When the PMU is enabled it is valid to have unhandled nmis, two
events could trigger 'simultaneously' raising two back-to-back
NMIs. If the first NMI handles both, the latter will be empty and daze
the CPU.

The solution to avoid an 'unknown nmi' massage in this case was simply
to stop the nmi handler chain when the PMU is enabled by stating
the nmi was handled. This has the drawback that a) we can not detect
unknown nmis anymore, and b) subsequent nmi handlers are not called.

This patch addresses this. Now, we check this unknown NMI if it could
be a PMU back-to-back NMI. Otherwise we pass it and let the kernel
handle the unknown nmi.

This is a debug log:

cpu #6, nmi #32333, skip_nmi #32330, handled = 1, time = 1934364430
cpu #6, nmi #32334, skip_nmi #32330, handled = 1, time = 1934704616
cpu #6, nmi #32335, skip_nmi #32336, handled = 2, time = 1936032320
cpu #6, nmi #32336, skip_nmi #32336, handled = 0, time = 1936034139
cpu #6, nmi #32337, skip_nmi #32336, handled = 1, time = 1936120100
cpu #6, nmi #32338, skip_nmi #32336, handled = 1, time = 1936404607
cpu #6, nmi #32339, skip_nmi #32336, handled = 1, time = 1937983416
cpu #6, nmi #32340, skip_nmi #32341, handled = 2, time = 1938201032
cpu #6, nmi #32341, skip_nmi #32341, handled = 0, time = 1938202830
cpu #6, nmi #32342, skip_nmi #32341, handled = 1, time = 1938443743
cpu #6, nmi #32343, skip_nmi #32341, handled = 1, time = 1939956552
cpu #6, nmi #32344, skip_nmi #32341, handled = 1, time = 1940073224
cpu #6, nmi #32345, skip_nmi #32341, handled = 1, time = 1940485677
cpu #6, nmi #32346, skip_nmi #32347, handled = 2, time = 1941947772
cpu #6, nmi #32347, skip_nmi #32347, handled = 1, time = 1941949818
cpu #6, nmi #32348, skip_nmi #32347, handled = 0, time = 1941951591
Uhhuh. NMI received for unknown reason 00 on CPU 6.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue

Deltas:

nmi #32334 340186
nmi #32335 1327704
nmi #32336 1819      <<<< back-to-back nmi [1]
nmi #32337 85961
nmi #32338 284507
nmi #32339 1578809
nmi #32340 217616
nmi #32341 1798      <<<< back-to-back nmi [2]
nmi #32342 240913
nmi #32343 1512809
nmi #32344 116672
nmi #32345 412453
nmi #32346 1462095   <<<< 1st nmi (standard) handling 2 counters
nmi #32347 2046      <<<< 2nd nmi (back-to-back) handling one counter
nmi #32348 1773      <<<< 3rd nmi (back-to-back) handling no counter! [3]

For  back-to-back nmi detection there are the following rules:

The PMU nmi handler was handling more than one counter and no
counter was handled in the subsequent nmi (see [1] and [2] above).

There is another case if there are two subsequent back-to-back nmis
[3]. The 2nd is detected as back-to-back because the first handled
more than one counter. If the second handles one counter and the 3rd
handles nothing, we drop the 3rd nmi because it could be a
back-to-back nmi.

Signed-off-by: Robert Richter <robert.richter@amd.com>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Tested-by: Don Zickus <dzickus@redhat.com>
LKML-Reference: <20100817152225.GQ26154@erda.amd.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 arch/x86/kernel/cpu/perf_event.c |   59 +++++++++++++++++++++++++++++--------
 1 files changed, 46 insertions(+), 13 deletions(-)

diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c
index f2da20f..dd2fceb 100644
--- a/arch/x86/kernel/cpu/perf_event.c
+++ b/arch/x86/kernel/cpu/perf_event.c
@@ -1154,7 +1154,7 @@ static int x86_pmu_handle_irq(struct pt_regs *regs)
 		/*
 		 * event overflow
 		 */
-		handled		= 1;
+		handled++;
 		data.period	= event->hw.last_period;
 
 		if (!x86_perf_event_set_period(event))
@@ -1200,12 +1200,20 @@ void perf_events_lapic_init(void)
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 }
 
+struct pmu_nmi_state {
+	unsigned int	marked;
+	int		handled;
+};
+
+static DEFINE_PER_CPU(struct pmu_nmi_state, nmi);
+
 static int __kprobes
 perf_event_nmi_handler(struct notifier_block *self,
 			 unsigned long cmd, void *__args)
 {
 	struct die_args *args = __args;
-	struct pt_regs *regs;
+	unsigned int this_nmi;
+	int handled;
 
 	if (!atomic_read(&active_events))
 		return NOTIFY_DONE;
@@ -1214,22 +1222,47 @@ perf_event_nmi_handler(struct notifier_block *self,
 	case DIE_NMI:
 	case DIE_NMI_IPI:
 		break;
-
+	case DIE_NMIUNKNOWN:
+		this_nmi = percpu_read(irq_stat.__nmi_count);
+		if (this_nmi != __get_cpu_var(nmi).marked)
+			/* let the kernel handle the unknown nmi */
+			return NOTIFY_DONE;
+		/*
+		 * This one is a PMU back-to-back nmi. Two events
+		 * trigger 'simultaneously' raising two back-to-back
+		 * NMIs. If the first NMI handles both, the latter
+		 * will be empty and daze the CPU. So, we drop it to
+		 * avoid false-positive 'unknown nmi' messages.
+		 */
+		return NOTIFY_STOP;
 	default:
 		return NOTIFY_DONE;
 	}
 
-	regs = args->regs;
-
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
-	/*
-	 * Can't rely on the handled return value to say it was our NMI, two
-	 * events could trigger 'simultaneously' raising two back-to-back NMIs.
-	 *
-	 * If the first NMI handles both, the latter will be empty and daze
-	 * the CPU.
-	 */
-	x86_pmu.handle_irq(regs);
+
+	handled = x86_pmu.handle_irq(args->regs);
+	if (!handled)
+		return NOTIFY_DONE;
+
+	this_nmi = percpu_read(irq_stat.__nmi_count);
+	if ((handled > 1) ||
+		/* the next nmi could be a back-to-back nmi */
+	    ((__get_cpu_var(nmi).marked == this_nmi) &&
+	     (__get_cpu_var(nmi).handled > 1))) {
+		/*
+		 * We could have two subsequent back-to-back nmis: The
+		 * first handles more than one counter, the 2nd
+		 * handles only one counter and the 3rd handles no
+		 * counter.
+		 *
+		 * This is the 2nd nmi because the previous was
+		 * handling more than one counter. We will mark the
+		 * next (3rd) and then drop it if unhandled.
+		 */
+		__get_cpu_var(nmi).marked	= this_nmi + 1;
+		__get_cpu_var(nmi).handled	= handled;
+	}
 
 	return NOTIFY_STOP;
 }