public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/2] tracing/tlb/x85: Fix splat of calling RCU trace code on offline CPU
@ 2015-02-06 20:06 Steven Rostedt
  2015-02-06 20:06 ` [PATCH 1/2] tracing: Add condition check to RCU lockdep checks Steven Rostedt
                   ` (2 more replies)
  0 siblings, 3 replies; 37+ messages in thread
From: Steven Rostedt @ 2015-02-06 20:06 UTC (permalink / raw)
  To: linux-kernel
  Cc: Paul E. McKenney, Dave Hansen, Rafael J. Wysocki, linux-next,
	Stephen Rothwell, Kristen Carlson Accardi, H. Peter Anvin,
	Rik van Riel, Mel Gorman, Andrew Morton

Paul,

I found a much better fix than adding the rcu_nocheck(). Simply have the
rcu check inside the condition check as well. This way the rcu splat
will only happen if the condition is set too. The condition doesn't need
the tracepoint enabled.

Now I'm thinking that I should push the first patch through my tree as it
only touches tracing. The second patch you can freely take.

Neither patch really depends on the other, but both patches are required
to make the splat go away. If Sedat could test these patches together,
and give his tested-by tag, that would be great. I'll run my patch through
my full series of tests and then push to linux next. You could take the second
patch and push that through your tree (linux-next). When both arrive, the
bug will be fixed. The two do not need to come in together.

Thoughts?

-- Steve


Steven Rostedt (Red Hat) (2):
      tracing: Add condition check to RCU lockdep checks
      x86/tbl/trace: Do not trace on CPU that is offline

----
 include/linux/tracepoint.h | 2 +-
 include/trace/events/tlb.h | 4 +++-
 2 files changed, 4 insertions(+), 2 deletions(-)

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [PATCH 0/2] [GIT PULL] tracing: Fix tlb_flush TP called in RCU ignored location
@ 2015-02-08  0:58 Steven Rostedt
  2015-02-08  0:58 ` [PATCH 1/2] tracing: Add condition check to RCU lockdep checks Steven Rostedt
  0 siblings, 1 reply; 37+ messages in thread
From: Steven Rostedt @ 2015-02-08  0:58 UTC (permalink / raw)
  To: linux-kernel; +Cc: Linus Torvalds, Ingo Molnar, Andrew Morton


Linus,

During testing Sedat Dilek hit a "suspicious RCU usage" splat that pointed
out a real bug. During suspend and resume the tlb_flush tracepoint is
called when the CPU is going offline. As the CPU has been noted as offline,
RCU is ignoring that CPU, which means that it can not use RCU protected
locks. When tracepoints are activated, they require RCU locking, and
if RCU is ignoring a CPU that runs a tracepoint, there is a chance that
the tracepoint could cause corruption.

The solution was to change the tracepoint into a TRACE_EVENT_CONDITION()
which allows us to check a condition to determine if the tracepoint
should be called or not. If the condition is not met, the rcu protected
code will not be executed. By adding the condition
"cpu_online(smp_processor_id())", this will prevent the RCU protected
code from being executed if the CPU is marked offline.

After adding this, another bug was discovered. As RCU checks rcu callers,
if a rcu call is not done, there is no check (obviously). We found that
tracepoints could be added in RCU ignored locations and not have lockdep
complain until the tracepoint is activated. This missed places where
tracepoints were added in places they should not have been. To fix this,
code was added in 3.18 that if lockdep is enabled, any tracepoint will
still call the rcu checks even if the tracepoint is not enabled. The bug
here, is that the check does not take the CONDITION into account. As the
condition may prevent tracepoints from being activated in RCU ignored
areas (as the one patch does), we get false positives when we enable
lockdep and hit a tracepoint that the condition prevents it from being
called in a RCU ignored location. The fix for this is to add the
CONDITION to the rcu checks, even if the tracepoint is not enabled.

Please pull the latest trace-fixes-v3.19-rc7 tree, which can be found at:


  git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git
trace-fixes-v3.19-rc7

Tag SHA1: 8c1c683d0b8af59da6b65976bb7f2a2f1cac7345
Head SHA1: 6c8465a82a605bc692304bab42703017dcfff013


Steven Rostedt (Red Hat) (2):
      tracing: Add condition check to RCU lockdep checks
      x86/tlb/trace: Do not trace on CPU that is offline

----
 include/linux/tracepoint.h | 2 +-
 include/trace/events/tlb.h | 4 +++-
 2 files changed, 4 insertions(+), 2 deletions(-)

^ permalink raw reply	[flat|nested] 37+ messages in thread
end of thread, other threads:[~2015-02-08  0:59 UTC | newest]

Thread overview: 37+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-02-06 20:06 [PATCH 0/2] tracing/tlb/x85: Fix splat of calling RCU trace code on offline CPU Steven Rostedt
2015-02-06 20:06 ` [PATCH 1/2] tracing: Add condition check to RCU lockdep checks Steven Rostedt
2015-02-06 20:11   ` Steven Rostedt
2015-02-06 20:13   ` [PATCH 1/2 v2] " Steven Rostedt
2015-02-06 20:06 ` [PATCH 2/2] x86/tbl/trace: Do not trace on CPU that is offline Steven Rostedt
2015-02-06 23:27   ` Paul E. McKenney
2015-02-07  4:02     ` Steven Rostedt
2015-02-07  8:01       ` Sedat Dilek
2015-02-07 15:20         ` Steven Rostedt
2015-02-07 19:50           ` Sedat Dilek
2015-02-07 20:09           ` Paul E. McKenney
2015-02-07 20:14             ` Sedat Dilek
2015-02-07 21:52             ` Steven Rostedt
2015-02-07 22:14               ` Paul E. McKenney
2015-02-07 23:01                 ` Sedat Dilek
2015-02-07 23:48               ` Dave Hansen
2015-02-07  8:13       ` Sedat Dilek
2015-02-07 15:22         ` Steven Rostedt
2015-02-06 21:07 ` [PATCH 0/2] tracing/tlb/x85: Fix splat of calling RCU trace code on offline CPU Sedat Dilek
2015-02-06 21:18   ` Steven Rostedt
2015-02-06 21:19     ` Steven Rostedt
2015-02-06 21:23       ` Sedat Dilek
2015-02-06 21:27         ` Steven Rostedt
2015-02-06 21:24       ` Steven Rostedt
2015-02-06 21:29         ` Sedat Dilek
2015-02-06 21:21     ` Sedat Dilek
2015-02-06 21:28       ` Steven Rostedt
2015-02-06 21:33       ` Steven Rostedt
2015-02-06 21:38   ` Paul E. McKenney
2015-02-06 22:13     ` Sedat Dilek
2015-02-06 22:35       ` Steven Rostedt
2015-02-06 22:48         ` Paul E. McKenney
2015-02-06 22:51           ` Sedat Dilek
2015-02-06 23:02             ` Sedat Dilek
2015-02-06 23:04             ` Paul E. McKenney
2015-02-06 23:04           ` Steven Rostedt
  -- strict thread matches above, loose matches on Subject: below --
2015-02-08  0:58 [PATCH 0/2] [GIT PULL] tracing: Fix tlb_flush TP called in RCU ignored location Steven Rostedt
2015-02-08  0:58 ` [PATCH 1/2] tracing: Add condition check to RCU lockdep checks Steven Rostedt

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox