Re: [PATCH] rcu/tree: Add a trace event for RCU stall warnings

["Paul E. McKenney" <paulmck@kernel.org>]

On Thu, Feb 18, 2021 at 09:17:01AM -0500, Mathieu Desnoyers wrote:
> ----- On Feb 18, 2021, at 7:52 AM, Sangmoon Kim sangmoon.kim@samsung.com wrote:
> 
> >> -----Original Message-----
> >> From: neeraju=codeaurora.org@mg.codeaurora.org
> >> <neeraju=codeaurora.org@mg.codeaurora.org>
> >> Sent: Thursday, February 18, 2021 3:18 AM
> >> 
> >> Hi Sangmoon,
> >> 
> >> On 2/17/2021 7:19 PM, Sangmoon Kim wrote:
> >> >> -----Original Message-----
> >> >> From: Paul E. McKenney <paulmck@kernel.org>
> >> >> Sent: Wednesday, February 17, 2021 2:50 AM
> >> >>
> >> >> On Mon, Feb 15, 2021 at 05:53:25PM +0900, Sangmoon Kim wrote:
> >> >>> The event allows us to trace the RCU stall when
> >> >>> sysctl_panic_on_rcu_stall is disabled.
> >> >>>
> >> >>> The first parameter is the name of RCU flavour like other trace
> >> >>> events. The second one shows us which function detected stalls.
> >> >>>
> >> >>> The RCU stall is mainly caused by external factors such as interrupt
> >> >>> handling or task scheduling or something else. Therefore, this event
> >> >>> uses TRACE_EVENT macro, not dedicated one, so that someone interested
> >> >>> in the RCU stall can use it without CONFIG_RCU_TRACE.
> >> >>>
> >> >>> Signed-off-by: Sangmoon Kim <sangmoon.kim@samsung.com>
> >> >>
> >> >> The patch looks plausible, but I have to ask...  Why not instead just
> >> >> get the existing information out of the console log?
> >> >>
> >> >> 							Thanx, Paul
> >> >
> >> > This can provide a trigger point for the RCU stall warning.
> >> > If a module in the kernel wants to trace the stall for debugging purposes,
> >> > there is a cost of continuing to parse the console log.
> >> > This tracepoint is useful because it is hard to pay these costs
> >> > especially on mobile devices.
> >> >
> >> > Thanks,
> >> > Sangmoon
> >> >
> >> 
> >> So, the idea here is to register to these trace events from kernel
> >> module and use that for debugging? Just curious what debugging action
> >> module does on these traces, as they have limited information
> >> about the stall, compared to console stall warnings, which gives a
> >> much more detailed information about stall.
> >> 
> >> 
> >> Thanks
> >> Neeraj
> > 
> > Hi Neeraj,
> > 
> > Yes, a module can log the stall occurence using the trace, although
> > there is no detailed information. If the kernel panic occurs for some
> > reasons, the debugging report generated by the module can include that
> > RCU stall warning has occurred before.
> > 
> > In addition, it's just an idea now, when a trace event happens, the
> > module can store the console log including detailed information, or may
> > also obtain CPU/task information by parsing the console log.
> 
> Adding a new tracepoint is not just about what is extracted by this specific
> tracepoint, but rather how it can be analyzed when combined with all other relevant
> tracepoints.
> 
> For instance, if we have this added RCU stall warning added, here is how it can be
> used with the upcoming LTTng 2.13, which implements the "event notification" triggers
> feature:
> 
> 1) Setup "flight recorder" (snapshot) tracing to trace into a circular ring buffer,
>    enabling the following tracepoints:
>    - kernel activity (meaning all other RCU event, scheduling, irq, workqueues, ...),
>    - this new RCU stall warning event.
> 
> 2) Add a "callstack-kernel" context to the RCU stall warning event. This will sample
>    the kernel stack when the event is hit. This will provide information similar to
>    the stack trace gathered into the console log on OOPS.
> 
> 3) Enable a trigger waiting on the RCU stall warning tracepoint to be hit. On this
>    trigger, actions can be associated, such as capturing a snapshot or waking up
>    an external user-space process to perform specific actions.
> 
> So you end up with a snapshot containing the sequence of events leading to the
> RCU stall warning, with a kernel stack trace of the context causing the stall
> warning to be emitted.
> 
> I would argue that this information is more complete than just the stack trace
> extracted through the console log.

I am not so sure about that.  RCU CPU stall warnings dump quite a bit more
than a stack trace to the console.  Which is why I am concerned about the
proverbial camel's nose in the tent.  ;-)

So Sangmoon, what is it that you really need for this to be useful to you?

Or am I missing your point?  (Either Mathieu's or Sangmoon's.)

							Thanx, Paul