Re: Query: Regarding Notifier chain callback debugging or profiling

[Greg KH <gregkh@linuxfoundation.org>]

On Mon, Feb 10, 2020 at 05:26:16PM +0530, Gaurav Kohli wrote:
> Hi,
> 
> In Linux kernel, everywhere we are using notification chains to notify for
> any kernel events, But we don't have any debugging or profiling mechanism to
> know which callback is taking time or currently we are stuck on which call
> back(without dumps it is difficult to say for last problem)

Callbacks are a mess, I agree.

> Below are the few ways, which we can implement to profile callback on need
> basis:
> 
> 1) Use trace event before and after callback:
> 
> static int notifier_call_chain(struct notifier_block **nl,
>                                unsigned long val, void *v,
>                                int nr_to_call, int *nr_calls)
> {
>         int ret = NOTIFY_DONE;
>         struct notifier_block *nb, *next_nb;
> 
> 
> +		trace_event for entry of callback
>                 ret = nb->notifier_call(nb, val, v);
> +		trace_event for exit of callback

Ick.

>         }
>         return ret;
> }
> 
> 2) Or use pr_debug instead of trace_event
> 
> 3) Both of the above approach has certain problems, like it will dump
> callback for each notifier chain, which might flood trace buffer or dmesg.
> 
> So we can use bool variable to control that and dump the required
> notification chain only.
> 
> Some thing like below we can use:
> 
>  struct srcu_notifier_head {
>         struct mutex mutex;
>         struct srcu_struct srcu;
>         struct notifier_block __rcu *head;
> +       bool debug_callback;
>  };
> 
> 
>  static int notifier_call_chain(struct notifier_block **nl,
>                                unsigned long val, void *v,
> -                              int nr_to_call, int *nr_calls)
> +                              int nr_to_call, int *nr_calls, bool
> debug_callback)
>  {
>         int ret = NOTIFY_DONE;
>         struct notifier_block *nb, *next_nb;
> @@ -526,6 +526,7 @@ void srcu_init_notifier_head(struct srcu_notifier_head
> *nh)
>         if (init_srcu_struct(&nh->srcu) < 0)
>                 BUG();
>         nh->head = NULL;
> +       nh->debug_callback = false; -> by default it would be false for
> every notifier chain.
> 
> 4) we can also think of something pre and post function, before and after
> each callback, And we can enable only for those who wants to profile.
> 
> Please let us what approach we can use, or please suggest some debugging
> mechanism for the same.

Why not just pay attention to the specific notifier you want?  Trace
when the specific blocking_notifier_call_chain() is called.

What specific notifier call chain is causing you problems that you need
to debug?

thanks,

greg k-h