[PATCH] tracing: generate RCU warnings even when tracepoints are disabled

[PATCH] tracing: generate RCU warnings even when tracepoints are disabled

[Dave Hansen]

From: Dave Hansen <dave.hansen@linux.intel.com>

Dave Jones reported seeing a bug from one of my TLB tracepoints:

	http://lkml.kernel.org/r/20140806181801.GA4605@redhat.com

I've been running these patches for months and never saw this.
But, a big chunk of my testing, especially with all the debugging
enabled, was in a vm where intel_idle doesn't work.  On the
systems where I was using intel_idle, I never had lockdep enabled
and this tracepoint on at the same time.

This patch ensures that whenever we have lockdep available, we do
_some_ RCU activity at the site of the tracepoint, despite
whether the tracepoint's condition matches or even if the
tracepoint itself is completely disabled.  This is a bit of a
hack, but it is pretty self-contained.

I confirmed that with this patch plus lockdep I get the same
splat as Dave Jones did, but without enabling the tracepoint
explicitly.

Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Dave Jones <davej@redhat.com>,
Cc: paulmck@linux.vnet.ibm.com
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ingo Molnar <mingo@redhat.com>

---

 b/include/linux/tracepoint.h |   11 +++++++++++
 1 file changed, 11 insertions(+)

diff -puN include/linux/tracepoint.h~easier-rcu-splat include/linux/tracepoint.h
--- a/include/linux/tracepoint.h~easier-rcu-splat	2014-08-07 10:29:32.701217956 -0700
+++ b/include/linux/tracepoint.h	2014-08-07 10:40:35.244627014 -0700
@@ -157,6 +157,12 @@ extern void syscall_unregfunc(void);
  * Make sure the alignment of the structure in the __tracepoints section will
  * not add unwanted padding between the beginning of the section and the
  * structure. Force alignment to the same alignment as the section start.
+ *
+ * When lockdep is enabled, we make sure to always do the RCU portions of
+ * the tracepoint code, regardless of whether tracing is on or we match the
+ * condition.  This lets us find RCU issues triggered with tracepoints even
+ * when this tracepoint is off.  This code has no purpose other than poking
+ * RCU a bit.
  */
 #define __DECLARE_TRACE(name, proto, args, cond, data_proto, data_args) \
 	extern struct tracepoint __tracepoint_##name;			\
@@ -167,6 +173,11 @@ extern void syscall_unregfunc(void);
 				TP_PROTO(data_proto),			\
 				TP_ARGS(data_args),			\
 				TP_CONDITION(cond),,);			\
+		if (IS_ENABLED(CONFIG_LOCKDEP)) {			\
+			rcu_read_lock_sched_notrace();			\
+			rcu_dereference_sched(__tracepoint_##name.funcs);\
+			rcu_read_unlock_sched_notrace();		\
+		}							\
 	}								\
 	__DECLARE_TRACE_RCU(name, PARAMS(proto), PARAMS(args),		\
 		PARAMS(cond), PARAMS(data_proto), PARAMS(data_args))	\
_

Re: [PATCH] tracing: generate RCU warnings even when tracepoints are disabled

[Paul E. McKenney]

On Thu, Aug 07, 2014 at 10:52:04AM -0700, Dave Hansen wrote:
> 
> From: Dave Hansen <dave.hansen@linux.intel.com>
> 
> Dave Jones reported seeing a bug from one of my TLB tracepoints:
> 
> 	http://lkml.kernel.org/r/20140806181801.GA4605@redhat.com
> 
> I've been running these patches for months and never saw this.
> But, a big chunk of my testing, especially with all the debugging
> enabled, was in a vm where intel_idle doesn't work.  On the
> systems where I was using intel_idle, I never had lockdep enabled
> and this tracepoint on at the same time.
> 
> This patch ensures that whenever we have lockdep available, we do
> _some_ RCU activity at the site of the tracepoint, despite
> whether the tracepoint's condition matches or even if the
> tracepoint itself is completely disabled.  This is a bit of a
> hack, but it is pretty self-contained.
> 
> I confirmed that with this patch plus lockdep I get the same
> splat as Dave Jones did, but without enabling the tracepoint
> explicitly.
> 
> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
> Cc: Dave Jones <davej@redhat.com>,
> Cc: paulmck@linux.vnet.ibm.com
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Ingo Molnar <mingo@redhat.com>

Looks good to me, but I must defer to Steven on this one.

							Thanx, Paul

> ---
> 
>  b/include/linux/tracepoint.h |   11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff -puN include/linux/tracepoint.h~easier-rcu-splat include/linux/tracepoint.h
> --- a/include/linux/tracepoint.h~easier-rcu-splat	2014-08-07 10:29:32.701217956 -0700
> +++ b/include/linux/tracepoint.h	2014-08-07 10:40:35.244627014 -0700
> @@ -157,6 +157,12 @@ extern void syscall_unregfunc(void);
>   * Make sure the alignment of the structure in the __tracepoints section will
>   * not add unwanted padding between the beginning of the section and the
>   * structure. Force alignment to the same alignment as the section start.
> + *
> + * When lockdep is enabled, we make sure to always do the RCU portions of
> + * the tracepoint code, regardless of whether tracing is on or we match the
> + * condition.  This lets us find RCU issues triggered with tracepoints even
> + * when this tracepoint is off.  This code has no purpose other than poking
> + * RCU a bit.
>   */
>  #define __DECLARE_TRACE(name, proto, args, cond, data_proto, data_args) \
>  	extern struct tracepoint __tracepoint_##name;			\
> @@ -167,6 +173,11 @@ extern void syscall_unregfunc(void);
>  				TP_PROTO(data_proto),			\
>  				TP_ARGS(data_args),			\
>  				TP_CONDITION(cond),,);			\
> +		if (IS_ENABLED(CONFIG_LOCKDEP)) {			\
> +			rcu_read_lock_sched_notrace();			\
> +			rcu_dereference_sched(__tracepoint_##name.funcs);\
> +			rcu_read_unlock_sched_notrace();		\
> +		}							\
>  	}								\
>  	__DECLARE_TRACE_RCU(name, PARAMS(proto), PARAMS(args),		\
>  		PARAMS(cond), PARAMS(data_proto), PARAMS(data_args))	\
> _
> 

Re: [PATCH] tracing: generate RCU warnings even when tracepoints are disabled

[Steven Rostedt]

On Thu, 7 Aug 2014 11:13:56 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

> On Thu, Aug 07, 2014 at 10:52:04AM -0700, Dave Hansen wrote:
> > 
> > From: Dave Hansen <dave.hansen@linux.intel.com>
> > 
> > Dave Jones reported seeing a bug from one of my TLB tracepoints:
> > 
> > 	http://lkml.kernel.org/r/20140806181801.GA4605@redhat.com
> > 
> > I've been running these patches for months and never saw this.
> > But, a big chunk of my testing, especially with all the debugging
> > enabled, was in a vm where intel_idle doesn't work.  On the
> > systems where I was using intel_idle, I never had lockdep enabled
> > and this tracepoint on at the same time.
> > 
> > This patch ensures that whenever we have lockdep available, we do
> > _some_ RCU activity at the site of the tracepoint, despite
> > whether the tracepoint's condition matches or even if the
> > tracepoint itself is completely disabled.  This is a bit of a
> > hack, but it is pretty self-contained.
> > 
> > I confirmed that with this patch plus lockdep I get the same
> > splat as Dave Jones did, but without enabling the tracepoint
> > explicitly.
> > 
> > Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
> > Cc: Dave Jones <davej@redhat.com>,
> > Cc: paulmck@linux.vnet.ibm.com
> > Cc: Steven Rostedt <rostedt@goodmis.org>
> > Cc: Ingo Molnar <mingo@redhat.com>
> 
> Looks good to me, but I must defer to Steven on this one.
> 

Looks fine. I can add it to my 3.18 queue.

Paul, want to send me an "Acked-by"?

Thanks,

-- Steve

Re: [PATCH] tracing: generate RCU warnings even when tracepoints are disabled

[Paul E. McKenney]

On Fri, Aug 08, 2014 at 02:30:22PM -0400, Steven Rostedt wrote:
> On Thu, 7 Aug 2014 11:13:56 -0700
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> 
> > On Thu, Aug 07, 2014 at 10:52:04AM -0700, Dave Hansen wrote:
> > > 
> > > From: Dave Hansen <dave.hansen@linux.intel.com>
> > > 
> > > Dave Jones reported seeing a bug from one of my TLB tracepoints:
> > > 
> > > 	http://lkml.kernel.org/r/20140806181801.GA4605@redhat.com
> > > 
> > > I've been running these patches for months and never saw this.
> > > But, a big chunk of my testing, especially with all the debugging
> > > enabled, was in a vm where intel_idle doesn't work.  On the
> > > systems where I was using intel_idle, I never had lockdep enabled
> > > and this tracepoint on at the same time.
> > > 
> > > This patch ensures that whenever we have lockdep available, we do
> > > _some_ RCU activity at the site of the tracepoint, despite
> > > whether the tracepoint's condition matches or even if the
> > > tracepoint itself is completely disabled.  This is a bit of a
> > > hack, but it is pretty self-contained.
> > > 
> > > I confirmed that with this patch plus lockdep I get the same
> > > splat as Dave Jones did, but without enabling the tracepoint
> > > explicitly.
> > > 
> > > Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
> > > Cc: Dave Jones <davej@redhat.com>,
> > > Cc: paulmck@linux.vnet.ibm.com
> > > Cc: Steven Rostedt <rostedt@goodmis.org>
> > > Cc: Ingo Molnar <mingo@redhat.com>
> > 
> > Looks good to me, but I must defer to Steven on this one.
> 
> Looks fine. I can add it to my 3.18 queue.
> 
> Paul, want to send me an "Acked-by"?

Here you go:

Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

							Thanx, Paul