From: Frederic Weisbecker <fweisbec@gmail.com>
To: Ingo Molnar <mingo@elte.hu>
Cc: Steven Rostedt <rostedt@goodmis.org>,
linux-kernel@vger.kernel.org,
Thomas Gleixner <tglx@linutronix.de>,
Peter Zijlstra <peterz@infradead.org>
Subject: Re: [PATCH 0/5] [GIT PULL] updates for tip/tracing/ftrace
Date: Fri, 20 Mar 2009 18:57:59 +0100 [thread overview]
Message-ID: <20090320175758.GA5936@nowhere> (raw)
In-Reply-To: <20090320170548.GC5937@nowhere>
On Fri, Mar 20, 2009 at 06:05:54PM +0100, Frederic Weisbecker wrote:
> On Thu, Mar 19, 2009 at 08:33:57AM +0100, Ingo Molnar wrote:
> >
> > * Ingo Molnar <mingo@elte.hu> wrote:
> >
> > > these latest ftrace changes caused a lockup on a -tip testsystem:
> >
> > Note, even with Rusty's offstack-cpumask fix applied (in latest
> > -tip), i can reproduce a lockup:
> >
> > [ 4.132872] calling init_mmio_trace+0x0/0x12 @ 1
> > [ 4.137551] initcall init_mmio_trace+0x0/0x12 returned 0 after 10 usecs
> > [ 4.144149] calling init_graph_trace+0x0/0x12 @ 1
> > [ 4.148913] Testing tracer function_graph: <3>INFO: RCU detected CPU 0 stall (t=4294680313/10000 jiffies)
> > [ 18.620342] Pid: 858, comm: kstop/0 Not tainted 2.6.29-rc8-tip-02638-g1c002f5-dirty #3067
> > [ 18.620342] Call Trace:
> > [ 18.620342] <IRQ> [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff8021b2d5>] print_context_stack+0xbc/0x11e
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff80219c2e>] dump_trace+0x221/0x271
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff8021b180>] show_trace_log_lvl+0x51/0x5e
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff8021b1a2>] show_trace+0x15/0x17
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff8021b505>] dump_stack+0x77/0x82
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff802e343b>] print_cpu_stall+0x40/0x9b
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff802e384f>] check_cpu_stall+0x11e/0x15c
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff802e38a4>] __rcu_pending+0x17/0xff
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff802e39b8>] rcu_pending+0x2c/0x5e
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff8028b89a>] update_process_times+0x3c/0x76
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff802a901e>] tick_periodic+0x7a/0x86
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff802a9050>] tick_handle_periodic+0x26/0x8a
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff802a97f8>] tick_do_broadcast+0x1d4/0x23a
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff802a9929>] tick_do_periodic_broadcast+0x4c/0x5d
> > [ 18.620342] [<ffffffff802e28b9>] ? rcu_torture_timer+0x0/0x12b
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff802a994e>] tick_handle_periodic_broadcast+0x14/0x75
> > [ 18.620342] [<ffffffff80214c3d>] return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff80219f0f>] timer_interrupt+0x1f/0x26
> > [ 18.620342] [<ffffffff811f1a6d>] ? _spin_unlock_irq+0x9/0x3b
> > [ 18.620342] [<ffffffff811f1a9b>] ? _spin_unlock_irq+0x37/0x3b
> > [ 18.620342] [<ffffffff80214c3d>] ? return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff802dd04c>] handle_IRQ_event+0x167/0x230
> > [ 18.620342] [<ffffffff8025e8e4>] ? idle_cpu+0x4/0x33
> > [ 18.620342] [<ffffffff80215a10>] ? restore_args+0x0/0x30
> > [ 18.620342] [<ffffffff80284453>] ? __local_bh_disable+0x11/0x16c
> > [ 18.620342] [<ffffffff80214c3d>] ? return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff802dfc86>] handle_edge_irq+0x1bf/0x225
> > [ 18.620342] [<ffffffff80214c3d>] ? return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff80219305>] handle_irq+0x15d/0x171
> > [ 18.620342] [<ffffffff802195e9>] ? do_softirq+0x80/0x187
> > [ 18.620342] [<ffffffff80214c3d>] ? return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff811f2457>] do_IRQ+0x5f/0xfc
> > [ 18.620342] [<ffffffff80214c3d>] ? return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff80215953>] ret_from_intr+0x0/0x16
> > [ 18.620342] [<ffffffff80214c3d>] ? return_to_handler+0x0/0x73
> > [ 18.620342] [<ffffffff8028aa1d>] run_timer_softirq+0x1c1/0x2af
> > [ 18.620342] <EOI> [<ffffffff802ce8f0>] ? ack_state+0x4/0x45
> > [ 18.620342] [<ffffffff802ced22>] ? stop_cpu+0x1c5/0x1d1
> > [ 18.620342] [<ffffffff80294a29>] ? run_workqueue+0x1a0/0x2e2
> > [ 18.620342] [<ffffffff802949d6>] ? run_workqueue+0x14d/0x2e2
> > [ 18.620342] [<ffffffff802ceb5d>] ? stop_cpu+0x0/0x1d1
> > [ 18.620342] [<ffffffff826b4140>] ? early_idt_handler+0x0/0x73
> > [ 18.620342] [<ffffffff80295d1b>] ? worker_thread+0x138/0x14c
> > [ 18.620342] [<ffffffff80299e63>] ? autoremove_wake_function+0x0/0x4f
> > [ 18.620342] [<ffffffff80295be3>] ? worker_thread+0x0/0x14c
> > [ 18.620342] [<ffffffff80299d20>] ? kthread+0x60/0xa0
> > [ 18.620342] [<ffffffff8021606a>] ? child_rip+0xa/0x20
> > [ 18.620342] [<ffffffff8026fd4b>] ? finish_task_switch+0x56/0x180
> > [ 18.620342] [<ffffffff80215a10>] ? restore_args+0x0/0x30
> > [ 18.620342] [<ffffffff80299cc0>] ? kthread+0x0/0xa0
> > [ 18.620342] [<ffffffff80216060>] ? child_rip+0x0/0x20
> > [ 64.758749] INFO: RCU detected CPU 0 stall (t=4294710313/40000 jiffies)
> > [ 64.758749] Pid: 858, comm: kstop/0 Not tainted 2.6.29-rc8-tip-02638-g1c002f5-dirty #3067
> >
> > (the hang is real - it lasted overnight without the system ever
> > managing to boot up.)
> >
> > Config (note, it's different from the config i sent last) and full
> > bootlog attached.
>
>
> Tested under x86-32 and can't reproduce...
> Hopefully this is not because I'm testing on UP.
>
Now I've tested on x86-64 SMP and still nothing.
Note that I've not only tested the selftests but also the function
graph tracer. But nothing weird appeared.
next prev parent reply other threads:[~2009-03-20 17:58 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-03-18 3:14 [PATCH 0/5] [GIT PULL] updates for tip/tracing/ftrace Steven Rostedt
2009-03-18 3:14 ` [PATCH 1/5] ring-buffer: add api to allow a tracer to change clock source Steven Rostedt
2009-03-18 3:14 ` [PATCH 2/5] tracing: add global-clock option to provide cross CPU clock to traces Steven Rostedt
2009-03-18 3:14 ` [PATCH 3/5] tracing: optimization of branch tracer Steven Rostedt
2009-03-18 3:14 ` [PATCH 4/5] tracing: make sched_switch stop/start light weight Steven Rostedt
2009-03-18 3:14 ` [PATCH 5/5] tracing: make power tracer start/stop methods lighter weight Steven Rostedt
2009-03-18 5:59 ` [PATCH 0/5] [GIT PULL] updates for tip/tracing/ftrace Ingo Molnar
2009-03-18 7:39 ` Ingo Molnar
2009-03-19 7:33 ` Ingo Molnar
2009-03-19 17:21 ` Steven Rostedt
2009-03-20 17:43 ` Paul E. McKenney
2009-03-20 18:36 ` Ingo Molnar
2009-03-20 18:38 ` Ingo Molnar
2009-03-20 19:19 ` Paul E. McKenney
2009-03-20 19:27 ` Ingo Molnar
2009-03-20 19:41 ` Paul E. McKenney
2009-03-20 19:46 ` Frederic Weisbecker
2009-03-20 19:54 ` Ingo Molnar
2009-03-20 20:48 ` Frederic Weisbecker
2009-03-20 21:05 ` Steven Rostedt
2009-03-21 10:01 ` Ingo Molnar
2009-03-21 16:58 ` Ingo Molnar
2009-03-21 17:25 ` Steven Rostedt
2009-03-21 19:07 ` Paul E. McKenney
2009-03-21 20:09 ` Ingo Molnar
2009-03-21 21:01 ` Paul E. McKenney
2009-03-22 14:24 ` Ingo Molnar
2009-03-22 15:06 ` Ingo Molnar
2009-03-22 17:02 ` Ingo Molnar
2009-03-22 18:33 ` Steven Rostedt
2009-03-22 19:52 ` Ingo Molnar
2009-03-23 18:44 ` Steven Rostedt
2009-03-21 17:32 ` Frederic Weisbecker
2009-03-21 17:44 ` Steven Rostedt
2009-03-21 17:53 ` Frederic Weisbecker
2009-03-21 18:17 ` Steven Rostedt
2009-03-21 20:03 ` Frederic Weisbecker
2009-03-21 18:18 ` Ingo Molnar
2009-03-21 20:09 ` Frederic Weisbecker
2009-03-21 20:46 ` Frederic Weisbecker
2009-03-22 19:41 ` Ingo Molnar
2009-03-22 20:41 ` Ingo Molnar
2009-03-20 21:39 ` Paul E. McKenney
2009-03-20 17:05 ` Frederic Weisbecker
2009-03-20 17:57 ` Frederic Weisbecker [this message]
2009-03-20 18:22 ` Steven Rostedt
2009-03-20 18:39 ` Frederic Weisbecker
2009-03-20 18:42 ` Ingo Molnar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20090320175758.GA5936@nowhere \
--to=fweisbec@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox