From: Sasha Levin <sasha.levin@oracle.com>
To: Dave Jones <davej@redhat.com>, Tejun Heo <tj@kernel.org>,
tglx@linutronix.de, Peter Zijlstra <peterz@infradead.org>,
LKML <linux-kernel@vger.kernel.org>,
trinity@vger.kernel.org
Subject: Re: timer: lockup in run_timer_softirq()
Date: Tue, 09 Jul 2013 19:09:51 -0400 [thread overview]
Message-ID: <51DC983F.2010409@oracle.com> (raw)
In-Reply-To: <20130709230316.GB13855@redhat.com>
On 07/09/2013 07:03 PM, Dave Jones wrote:
> On Tue, Jul 09, 2013 at 06:49:29PM -0400, Sasha Levin wrote:
> > On 07/09/2013 06:47 PM, Dave Jones wrote:
>
> > > > [ 2536.500130] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run:
> > > > 697182.008 msecs
> > >
> > > I've been reporting these (and other traces) for a while https://lkml.org/lkml/2013/7/5/185 [1]
> > > on bare-metal.
> >
> > Interesting. It's the first time I'm seeing these, but I haven't really changed
> > anything in my configuration.
> >
> > Are you also seeing the "NMI handler took too long" messages?
>
> Now that I look closer, one difference is you're seeing arch_trigger_all_cpu_backtrace_handler
> in that message. I'm seeing them from perf_event_nmi_handler.
perf huh? I also have this spew I'm currently working on, it seems related:
[ 1443.380407] ------------[ cut here ]------------
[ 1443.381713] WARNING: CPU: 20 PID: 49263 at kernel/lockdep.c:3552 check_flags+0x16b/0x220()
[ 1443.383988] DEBUG_LOCKS_WARN_ON(current->softirqs_enabled)
[ 1443.385452] Modules linked in:
[ 1443.386459] CPU: 20 PID: 49263 Comm: trinity-child50 Tainted: G W
3.10.0-next-20130709-sasha #3953
[ 1443.388735] 0000000000000de0 ffff880805e03ab8 ffffffff84191329 ffffffff8519f386
[ 1443.390082] ffff880805e03b08 ffff880805e03af8 ffffffff811279cc ffffffff8519f386
[ 1443.390082] 0000000000000000 ffff8807cf898000 ffffffff85a66940 0000000000000082
[ 1443.390082] Call Trace:
[ 1443.390082] <IRQ> [<ffffffff84191329>] dump_stack+0x52/0x87
[ 1443.390082] [<ffffffff811279cc>] warn_slowpath_common+0x8c/0xc0
[ 1443.390082] [<ffffffff81127ab6>] warn_slowpath_fmt+0x46/0x50
[ 1443.390082] [<ffffffff8419d649>] ? sub_preempt_count+0x9/0xf0
[ 1443.390082] [<ffffffff811a192b>] check_flags+0x16b/0x220
[ 1443.390082] [<ffffffff811a2ee7>] lock_is_held+0x77/0x110
[ 1443.390082] [<ffffffff8419d644>] ? sub_preempt_count+0x4/0xf0
[ 1443.390082] [<ffffffff8122dafe>] perf_tp_event+0xbe/0x450
[ 1443.390082] [<ffffffff8122de62>] ? perf_tp_event+0x422/0x450
[ 1443.390082] [<ffffffff8419d644>] ? sub_preempt_count+0x4/0xf0
[ 1443.390082] [<ffffffff812185f2>] perf_ftrace_function_call+0xc2/0xe0
[ 1443.390082] [<ffffffff811f8c78>] ? ftrace_ops_control_func+0xc8/0x140
[ 1443.427495] [<ffffffff8419d644>] ? sub_preempt_count+0x4/0xf0
[ 1443.427495] [<ffffffff81131917>] ? __local_bh_enable+0xc7/0xd0
[ 1443.427495] [<ffffffff811f8c78>] ftrace_ops_control_func+0xc8/0x140
[ 1443.427495] [<ffffffff841a1c3c>] ftrace_call+0x5/0x2f
[ 1443.427495] [<ffffffff841a1c3c>] ? ftrace_call+0x5/0x2f
[ 1443.427495] [<ffffffff81175b87>] ? vtime_account_irq_exit+0x67/0x80
[ 1443.427495] [<ffffffff8419d649>] ? sub_preempt_count+0x9/0xf0
[ 1443.427495] [<ffffffff8419d649>] ? sub_preempt_count+0x9/0xf0
[ 1443.427495] [<ffffffff8113185e>] ? __local_bh_enable+0xe/0xd0
[ 1443.427495] [<ffffffff8419d649>] ? sub_preempt_count+0x9/0xf0
[ 1443.427495] [<ffffffff81131917>] __local_bh_enable+0xc7/0xd0
[ 1443.427495] [<ffffffff81132d07>] __do_softirq+0x447/0x4d0
[ 1443.427495] [<ffffffff8419d649>] ? sub_preempt_count+0x9/0xf0
[ 1443.427495] [<ffffffff81132ed6>] irq_exit+0x86/0x120
[ 1443.427495] [<ffffffff841a43ea>] smp_apic_timer_interrupt+0x4a/0x60
[ 1443.427495] [<ffffffff841a2d32>] apic_timer_interrupt+0x72/0x80
[ 1443.427495] <EOI> [<ffffffff81130542>] ? do_setitimer+0x242/0x2a0
[ 1443.427495] [<ffffffff8419871c>] ? _raw_spin_unlock_irq+0x4c/0x80
[ 1443.427495] [<ffffffff84198700>] ? _raw_spin_unlock_irq+0x30/0x80
[ 1443.427495] [<ffffffff81130542>] do_setitimer+0x242/0x2a0
[ 1443.427495] [<ffffffff811306fa>] alarm_setitimer+0x3a/0x70
[ 1443.427495] [<ffffffff8113b41e>] SyS_alarm+0xe/0x20
[ 1443.427495] [<ffffffff841a2240>] tracesys+0xdd/0xe2
[ 1443.427495] ---[ end trace e3b9a6b9c7462a51 ]---
next prev parent reply other threads:[~2013-07-09 23:09 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-07-09 22:35 timer: lockup in run_timer_softirq() Sasha Levin
2013-07-09 22:47 ` Dave Jones
2013-07-09 22:49 ` Sasha Levin
2013-07-09 23:03 ` Dave Jones
2013-07-09 23:09 ` Sasha Levin [this message]
2013-07-10 9:52 ` Peter Zijlstra
2013-07-10 12:27 ` Steven Rostedt
2013-07-10 12:42 ` Peter Zijlstra
2013-07-10 12:58 ` Steven Rostedt
2013-07-11 16:42 ` Peter Zijlstra
2013-07-11 16:55 ` Steven Rostedt
2013-07-11 16:59 ` Steven Rostedt
2013-07-11 17:11 ` Peter Zijlstra
2014-04-03 20:43 ` Sasha Levin
2013-07-10 9:54 ` Peter Zijlstra
2013-07-10 14:47 ` Sasha Levin
2013-07-11 10:02 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=51DC983F.2010409@oracle.com \
--to=sasha.levin@oracle.com \
--cc=davej@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
--cc=tj@kernel.org \
--cc=trinity@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.