From: Ingo Molnar <mingo@elte.hu>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: LKML <linux-kernel@vger.kernel.org>,
Thomas Gleixner <tglx@linutronix.de>,
Peter Zijlstra <peterz@infradead.org>,
Rusty Russell <rusty@rustcorp.com.au>,
akpm@osdl.org
Subject: Re: [PATCH -v2] ftrace: disable function tracing bringing up new CPU
Date: Wed, 18 Jun 2008 10:25:01 +0200 [thread overview]
Message-ID: <20080618082501.GE4135@elte.hu> (raw)
In-Reply-To: <Pine.LNX.4.58.0806051916190.16872@gandalf.stny.rr.com>
* Steven Rostedt <rostedt@goodmis.org> wrote:
> Peter Zijlstra found that taking down and bringing up a new CPU caused
> ftrace to crash the kernel. This was due to some arch calls that were
> being traced by the function tracer before the smp_processor_id was
> set up. Since the function tracer uses smp_processor_id it caused a
> triple fault.
>
> Instead of adding notrace all over the architecture code to prevent
> this problem, it is easier to simply disable the function tracer when
> bringing up a new CPU.
this patch brought lockdep trouble - see below.
Ingo
------------------->
calling pcibios_irq_init+0x0/0x449
PCI: Using IRQ router PIIX/ICH [8086/2440] at 0000:00:1f.0
=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.26-rc5-00070-g53ec472-dirty #697
-------------------------------------------------------
ftraced/27 is trying to acquire lock:
(&cpu_hotplug.lock){--..}, at: [<c0125d44>] get_online_cpus+0x31/0x43
but task is already holding lock:
(ftraced_lock){--..}, at: [<c014f6ad>] ftraced+0x47/0x136
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #2 (ftraced_lock){--..}:
[<c01412ff>] __lock_acquire+0x9f8/0xb5f
[<c0141872>] lock_acquire+0x6a/0x90
[<c06efb64>] mutex_lock_nested+0xdd/0x27a
[<c014f6ad>] ftraced+0x47/0x136
[<c0135820>] kthread+0x40/0x68
[<c0103c27>] kernel_thread_helper+0x7/0x10
[<ffffffff>] 0xffffffff
-> #1 (ftrace_sysctl_lock){--..}:
[<c01412ff>] __lock_acquire+0x9f8/0xb5f
[<c0141872>] lock_acquire+0x6a/0x90
[<c06efb64>] mutex_lock_nested+0xdd/0x27a
[<c014e7ea>] ftrace_disable+0xf/0x25
[<c06edb43>] _cpu_up+0x8a/0xf1
[<c06edbf3>] cpu_up+0x49/0x59
[<c0a3d599>] kernel_init+0x8e/0x264
[<c0103c27>] kernel_thread_helper+0x7/0x10
[<ffffffff>] 0xffffffff
-> #0 (&cpu_hotplug.lock){--..}:
[<c0141226>] __lock_acquire+0x91f/0xb5f
[<c0141872>] lock_acquire+0x6a/0x90
[<c06efb64>] mutex_lock_nested+0xdd/0x27a
[<c0125d44>] get_online_cpus+0x31/0x43
[<c0147c34>] stop_machine_run+0x16/0x3d
[<c014f00a>] ftrace_update_code+0x2f/0x3a
[<c014f6cc>] ftraced+0x66/0x136
[<c0135820>] kthread+0x40/0x68
[<c0103c27>] kernel_thread_helper+0x7/0x10
[<ffffffff>] 0xffffffff
other info that might help us debug this:
2 locks held by ftraced/27:
#0: (ftrace_sysctl_lock){--..}, at: [<c014f6a1>] ftraced+0x3b/0x136
#1: (ftraced_lock){--..}, at: [<c014f6ad>] ftraced+0x47/0x136
stack backtrace:
Pid: 27, comm: ftraced Not tainted 2.6.26-rc5-00070-g53ec472-dirty #697
[<c013f6b1>] print_circular_bug_tail+0x5b/0x66
[<c013ef59>] ? print_circular_bug_entry+0x39/0x43
[<c0141226>] __lock_acquire+0x91f/0xb5f
[<c0141872>] lock_acquire+0x6a/0x90
[<c0125d44>] ? get_online_cpus+0x31/0x43
[<c06efb64>] mutex_lock_nested+0xdd/0x27a
[<c0125d44>] ? get_online_cpus+0x31/0x43
[<c0125d44>] ? get_online_cpus+0x31/0x43
[<c014ec54>] ? __ftrace_update_code+0x0/0x12e
[<c0125d44>] get_online_cpus+0x31/0x43
[<c0147c34>] stop_machine_run+0x16/0x3d
[<c014f666>] ? ftraced+0x0/0x136
[<c014f00a>] ftrace_update_code+0x2f/0x3a
[<c014f6cc>] ftraced+0x66/0x136
[<c014f666>] ? ftraced+0x0/0x136
[<c0135820>] kthread+0x40/0x68
[<c01357e0>] ? kthread+0x0/0x68
[<c0103c27>] kernel_thread_helper+0x7/0x10
=======================
PCI->APIC IRQ transform: 0000:00:1f.2[D] -> IRQ 19
PCI->APIC IRQ transform: 0000:00:1f.3[B] -> IRQ 17
PCI->APIC IRQ transform: 0000:00:1f.4[C] -> IRQ 23
PCI->APIC IRQ transform: 0000:00:1f.5[B] -> IRQ 17
PCI->APIC IRQ transform: 0000:01:00.0[A] -> IRQ 22
PCI->APIC IRQ transform: 0000:03:04.0[A] -> IRQ 18
prev parent reply other threads:[~2008-06-18 8:25 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-06-05 20:54 [PATCH] ftrace: disable function tracing bringing up new CPU Steven Rostedt
2008-06-05 23:19 ` [PATCH -v2] " Steven Rostedt
2008-06-18 8:25 ` Ingo Molnar [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20080618082501.GE4135@elte.hu \
--to=mingo@elte.hu \
--cc=akpm@osdl.org \
--cc=linux-kernel@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=rusty@rustcorp.com.au \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox