public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
From: Steven Rostedt <rostedt@goodmis.org>
To: Ingo Molnar <mingo@elte.hu>,
	Peter Zijlstra <peterz@infradead.org>,
	Thomas Gleixner <tglx@linutronix.de>,
	Andrew Morton <akpm@linux-foundation.org>,
	linux-kernel@vger.kernel.org, Linus Torvalds <torvalds@osdl.org>,
	Arjan van de Ven <arjan@infradead.org>
Subject: [PATCH 0/5] ftrace: do not trace NMI handlers
Date: Tue, 29 Jul 2008 21:29:39 -0400	[thread overview]
Message-ID: <20080730012939.885172468@goodmis.org> (raw)


The dynamic ftrace code modifies code text at run time. Arjan informed
me that there is no safe way to modify code text on an SMP system when
the other CPUs might execute that same code. The reason has to do with
pipeline caches and CPUs might do funny things if the code being pushed
in the pipeline also happens to be modified at that same time. (Arjan
correct me if I'm wrong here).

We use kstop_machine to put the system into a UP like mode. This prevents
other CPUs from executing code while we modify it. Under stress testing
Ingo discovered that NMIs can cause the system to crash. This was due
to NMIs calling code that is being modified. Some boxes are more prone to
failure than others.

This series of patches performs two tasks:

 1) Add notrace to functions called by NMI, or simply remove the tracing
    completely from files that are primarily used by NMI.

 2) Add a warning when code that will be modified is called by an NMI.
    This also disables ftraced when it is detected, to prevent the
    race with the NMI and code modification from happneing.

The warning looks something like this:

--------------- cut here ---------------
WARNING: ftraced code called from NMI context lapic_wd_event+0xd/0x65
         Please report this to the ftrace maintainer.
         Disabling ftraced. Boot with ftrace_keep_on_nmi to not disable.
Pid: 0, comm: swapper Not tainted 2.6.26-tip #96

Call Trace:
 <NMI>  [<ffffffff8021c6d0>] ? lapic_wd_event+0xd/0x65
 [<ffffffff8027b9c1>] ftrace_record_ip+0xa3/0x357
 [<ffffffff8020c0f4>] mcount_call+0x5/0x31
 [<ffffffff8021c6d5>] ? lapic_wd_event+0x12/0x65
 [<ffffffff804b90d4>] nmi_watchdog_tick+0x21b/0x230
 [<ffffffff804b8487>] default_do_nmi+0x73/0x1e0
 [<ffffffff804b8a04>] do_nmi+0x64/0x91
 [<ffffffff804b80bf>] nmi+0x7f/0x80
 [<ffffffff80212c14>] ? default_idle+0x35/0x4f
 <<EOE>>  [<ffffffff8020ae42>] cpu_idle+0x8a/0xc9
 [<ffffffff804b15a6>] start_secondary+0x172/0x177

--------------- end cut here ---------------


This appears once when it is caught. We are hoping that this will not
appear often, and are running code to catch it as it does.

-- Steve


             reply	other threads:[~2008-07-30  1:44 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-07-30  1:29 Steven Rostedt [this message]
2008-07-30  1:29 ` [PATCH 1/5] ftrace: do not trace nmi callers in x86 Steven Rostedt
2008-07-30  1:29 ` [PATCH 2/5] ftrace: do not trace nmi callers in drivers Steven Rostedt
2008-07-30  1:29 ` [PATCH 3/5] ftrace: do not trace nmi callers in RCU Steven Rostedt
2008-07-30  1:29 ` [PATCH 4/5] ftrace: do not trace nmi callers in kernel directory Steven Rostedt
2008-07-30  1:29 ` [PATCH 5/5] ftrace: warn on NMI calling code that may be modified Steven Rostedt
2008-07-30  7:28 ` [PATCH 0/5] ftrace: do not trace NMI handlers Peter Zijlstra
2008-07-30 14:04   ` Mathieu Desnoyers
2008-07-30 14:23     ` Mathieu Desnoyers

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20080730012939.885172468@goodmis.org \
    --to=rostedt@goodmis.org \
    --cc=akpm@linux-foundation.org \
    --cc=arjan@infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@elte.hu \
    --cc=peterz@infradead.org \
    --cc=tglx@linutronix.de \
    --cc=torvalds@osdl.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox