All of lore.kernel.org
 help / color / mirror / Atom feed
From: Peter Zijlstra <peterz@infradead.org>
To: Frederic Weisbecker <fweisbec@gmail.com>
Cc: David Miller <davem@davemloft.net>,
	sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org,
	mingo@elte.hu, acme@redhat.com, paulus@samba.org,
	Mike Galbraith <efault@gmx.de>
Subject: Re: Random scheduler/unaligned accesses crashes with perf lock
Date: Tue, 06 Apr 2010 11:51:56 +0000	[thread overview]
Message-ID: <1270554716.1595.134.camel@laptop> (raw)
In-Reply-To: <20100406113830.GF5147@nowhere>

On Tue, 2010-04-06 at 13:38 +0200, Frederic Weisbecker wrote:
> On Tue, Apr 06, 2010 at 02:50:49AM -0700, David Miller wrote:
> > From: Frederic Weisbecker <fweisbec@gmail.com>
> > Date: Mon, 5 Apr 2010 21:40:58 +0200
> > 
> > > It happens without CONFIG_FUNCTION_TRACER as well (but it happens
> > > when the function tracer runs). And I hadn't your
> > > perf_arch_save_caller_regs() when I triggered this.
> > 
> > I figured out the problem, it's NMIs.  As soon as I disable all of the
> > NMI watchdog code, the problem goes away.
> > 
> > This is because some parts of the NMI interrupt handling path are not
> > marked with "notrace" and the various tracer code paths use
> > local_irq_disable() (either directly or indirectly) which doesn't work
> > with sparc64's NMI scheme.  These essentially turn NMIs back on in the
> > NMI handler before the NMI condition has been cleared, and thus we can
> > re-enter with another NMI interrupt.
> > 
> > We went through this for perf events, and we just made sure that
> > local_irq_{enable,disable}() never occurs in any of the code paths in
> > perf events that can be reached via the NMI interrupt handler.  (the
> > only one we had was sched_clock() and that was easily fixed)
> 
> 
> 
> That reminds me we have a new pair of local_irq_disable/enable
> in perf_event_task_output(), which path can be taken by hardware
> pmu events.
> 
> See this patch:
> 
> 8bb39f9aa068262732fe44b965d7a6eb5a5a7d67
> perf: Fix 'perf sched record' deadlock

ARGH.. yes

Also, I guess that should live in perf_output_lock/unlock() not in
perf_event_task_output().

Egads, how to fix that


WARNING: multiple messages have this Message-ID (diff)
From: Peter Zijlstra <peterz@infradead.org>
To: Frederic Weisbecker <fweisbec@gmail.com>
Cc: David Miller <davem@davemloft.net>,
	sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org,
	mingo@elte.hu, acme@redhat.com, paulus@samba.org,
	Mike Galbraith <efault@gmx.de>
Subject: Re: Random scheduler/unaligned accesses crashes with perf lock events on sparc 64
Date: Tue, 06 Apr 2010 13:51:56 +0200	[thread overview]
Message-ID: <1270554716.1595.134.camel@laptop> (raw)
In-Reply-To: <20100406113830.GF5147@nowhere>

On Tue, 2010-04-06 at 13:38 +0200, Frederic Weisbecker wrote:
> On Tue, Apr 06, 2010 at 02:50:49AM -0700, David Miller wrote:
> > From: Frederic Weisbecker <fweisbec@gmail.com>
> > Date: Mon, 5 Apr 2010 21:40:58 +0200
> > 
> > > It happens without CONFIG_FUNCTION_TRACER as well (but it happens
> > > when the function tracer runs). And I hadn't your
> > > perf_arch_save_caller_regs() when I triggered this.
> > 
> > I figured out the problem, it's NMIs.  As soon as I disable all of the
> > NMI watchdog code, the problem goes away.
> > 
> > This is because some parts of the NMI interrupt handling path are not
> > marked with "notrace" and the various tracer code paths use
> > local_irq_disable() (either directly or indirectly) which doesn't work
> > with sparc64's NMI scheme.  These essentially turn NMIs back on in the
> > NMI handler before the NMI condition has been cleared, and thus we can
> > re-enter with another NMI interrupt.
> > 
> > We went through this for perf events, and we just made sure that
> > local_irq_{enable,disable}() never occurs in any of the code paths in
> > perf events that can be reached via the NMI interrupt handler.  (the
> > only one we had was sched_clock() and that was easily fixed)
> 
> 
> 
> That reminds me we have a new pair of local_irq_disable/enable
> in perf_event_task_output(), which path can be taken by hardware
> pmu events.
> 
> See this patch:
> 
> 8bb39f9aa068262732fe44b965d7a6eb5a5a7d67
> perf: Fix 'perf sched record' deadlock

ARGH.. yes

Also, I guess that should live in perf_output_lock/unlock() not in
perf_event_task_output().

Egads, how to fix that


  reply	other threads:[~2010-04-06 11:51 UTC|newest]

Thread overview: 48+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2010-04-04 12:18 Random scheduler/unaligned accesses crashes with perf lock events Frederic Weisbecker
2010-04-04 12:18 ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 Frederic Weisbecker
2010-04-04 12:21 ` Random scheduler/unaligned accesses crashes with perf lock Frederic Weisbecker
2010-04-04 12:21   ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 Frederic Weisbecker
2010-04-05  1:00   ` Random scheduler/unaligned accesses crashes with perf lock David Miller
2010-04-05  1:00     ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 David Miller
2010-04-05  6:57     ` Random scheduler/unaligned accesses crashes with perf lock Frederic Weisbecker
2010-04-05  6:57       ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 Frederic Weisbecker
2010-04-05 19:22       ` Random scheduler/unaligned accesses crashes with perf lock David Miller
2010-04-05 19:22         ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 David Miller
2010-04-05 19:40         ` Random scheduler/unaligned accesses crashes with perf lock Frederic Weisbecker
2010-04-05 19:40           ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 Frederic Weisbecker
2010-04-05 20:46           ` Random scheduler/unaligned accesses crashes with perf lock David Miller
2010-04-05 20:46             ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 David Miller
2010-04-06  2:15           ` Random scheduler/unaligned accesses crashes with perf lock David Miller
2010-04-06  2:15             ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 David Miller
2010-04-06 13:41             ` Random scheduler/unaligned accesses crashes with perf lock Steven Rostedt
2010-04-06 13:41               ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 Steven Rostedt
2010-04-06 17:46               ` Random scheduler/unaligned accesses crashes with perf lock David Miller
2010-04-06 17:46                 ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 David Miller
2010-04-06 18:15                 ` Random scheduler/unaligned accesses crashes with perf lock Steven Rostedt
2010-04-06 18:15                   ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 Steven Rostedt
2010-04-06 21:17                   ` Random scheduler/unaligned accesses crashes with perf lock David Miller
2010-04-06 21:17                     ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 David Miller
2010-04-06  9:50           ` Random scheduler/unaligned accesses crashes with perf lock David Miller
2010-04-06  9:50             ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 David Miller
2010-04-06 10:19             ` Random scheduler/unaligned accesses crashes with perf lock Frederic Weisbecker
2010-04-06 10:19               ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 Frederic Weisbecker
2010-04-06 10:28               ` Random scheduler/unaligned accesses crashes with perf lock David Miller
2010-04-06 10:28                 ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 David Miller
2010-04-06 11:12             ` [RFC][PATCH] lockdep: WARN about local_irq_{en,dis}able in NMI Peter Zijlstra
2010-04-06 11:12               ` [RFC][PATCH] lockdep: WARN about local_irq_{en,dis}able in NMI context Peter Zijlstra
2010-04-06 11:13               ` [RFC][PATCH] lockdep: WARN about local_irq_{en,dis}able in NMI David Miller
2010-04-06 11:13                 ` [RFC][PATCH] lockdep: WARN about local_irq_{en,dis}able in NMI context David Miller
2010-04-06 11:20                 ` [RFC][PATCH] lockdep: WARN about local_irq_{en,dis}able in NMI Peter Zijlstra
2010-04-06 11:20                   ` [RFC][PATCH] lockdep: WARN about local_irq_{en,dis}able in NMI context Peter Zijlstra
2010-04-06 11:22                   ` [RFC][PATCH] lockdep: WARN about local_irq_{en,dis}able in NMI David Miller
2010-04-06 11:22                     ` [RFC][PATCH] lockdep: WARN about local_irq_{en,dis}able in NMI context David Miller
2010-04-06 11:38             ` Random scheduler/unaligned accesses crashes with perf lock Frederic Weisbecker
2010-04-06 11:38               ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 Frederic Weisbecker
2010-04-06 11:51               ` Peter Zijlstra [this message]
2010-04-06 11:51                 ` Peter Zijlstra
2010-04-06 12:54                 ` Random scheduler/unaligned accesses crashes with perf lock Mike Galbraith
2010-04-06 12:54                   ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 Mike Galbraith
2010-04-06 12:57                   ` Random scheduler/unaligned accesses crashes with perf lock Peter Zijlstra
2010-04-06 12:57                     ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 Peter Zijlstra
2010-04-06 18:04             ` Random scheduler/unaligned accesses crashes with perf lock Paul E. McKenney
2010-04-06 18:04               ` Random scheduler/unaligned accesses crashes with perf lock events on sparc 64 Paul E. McKenney

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1270554716.1595.134.camel@laptop \
    --to=peterz@infradead.org \
    --cc=acme@redhat.com \
    --cc=davem@davemloft.net \
    --cc=efault@gmx.de \
    --cc=fweisbec@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@elte.hu \
    --cc=paulus@samba.org \
    --cc=sparclinux@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.