[PATCH 6.1] softirq: Add trace points for tasklet entry/exit

[PATCH 6.1] softirq: Add trace points for tasklet entry/exit

[Sumanth Gavini]

commit f4bf3ca2e5cba655824b6e0893a98dfb33ed24e5 upstream.

Tasklets are supposed to finish their work quickly and should not block the
current running process, but it is not guaranteed that they do so.

Currently softirq_entry/exit can be used to analyse the total tasklets
execution time, but that's not helpful to track individual tasklets
execution time. That makes it hard to identify tasklet functions, which
take more time than expected.

Add tasklet_entry/exit trace point support to track individual tasklet
execution.

Trivial usage example:
   # echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_entry/enable
   # echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_exit/enable
   # cat /sys/kernel/debug/tracing/trace
 # tracer: nop
 #
 # entries-in-buffer/entries-written: 4/4   #P:4
 #
 #                                _-----=> irqs-off/BH-disabled
 #                               / _----=> need-resched
 #                              | / _---=> hardirq/softirq
 #                              || / _--=> preempt-depth
 #                              ||| / _-=> migrate-disable
 #                              |||| /     delay
 #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
 #              | |         |   |||||     |         |
           <idle>-0       [003] ..s1.   314.011428: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func
           <idle>-0       [003] ..s1.   314.011432: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func
           <idle>-0       [003] ..s1.   314.017369: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func
           <idle>-0       [003] ..s1.   314.017371: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func

Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Signed-off-by: J. Avila <elavila@google.com>
Signed-off-by: John Stultz <jstultz@google.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Link: https://lore.kernel.org/r/20230407230526.1685443-1-jstultz@google.com

[elavila: Port to android-mainline]
[jstultz: Rebased to upstream, cut unused trace points, added
 comments for the tracepoints, reworded commit]

Signed-off-by: Sumanth Gavini <sumanth.gavini@yahoo.com>
---
 include/trace/events/irq.h | 47 ++++++++++++++++++++++++++++++++++++++
 kernel/softirq.c           |  9 ++++++--
 2 files changed, 54 insertions(+), 2 deletions(-)

diff --git a/include/trace/events/irq.h b/include/trace/events/irq.h
index eeceafaaea4c..a07b4607b663 100644
--- a/include/trace/events/irq.h
+++ b/include/trace/events/irq.h
@@ -160,6 +160,53 @@ DEFINE_EVENT(softirq, softirq_raise,
 	TP_ARGS(vec_nr)
 );
 
+DECLARE_EVENT_CLASS(tasklet,
+
+	TP_PROTO(struct tasklet_struct *t, void *func),
+
+	TP_ARGS(t, func),
+
+	TP_STRUCT__entry(
+		__field(	void *,	tasklet)
+		__field(	void *,	func)
+	),
+
+	TP_fast_assign(
+		__entry->tasklet = t;
+		__entry->func = func;
+	),
+
+	TP_printk("tasklet=%ps function=%ps", __entry->tasklet, __entry->func)
+);
+
+/**
+ * tasklet_entry - called immediately before the tasklet is run
+ * @t: tasklet pointer
+ * @func: tasklet callback or function being run
+ *
+ * Used to find individual tasklet execution time
+ */
+DEFINE_EVENT(tasklet, tasklet_entry,
+
+	TP_PROTO(struct tasklet_struct *t, void *func),
+
+	TP_ARGS(t, func)
+);
+
+/**
+ * tasklet_exit - called immediately after the tasklet is run
+ * @t: tasklet pointer
+ * @func: tasklet callback or function being run
+ *
+ * Used to find individual tasklet execution time
+ */
+DEFINE_EVENT(tasklet, tasklet_exit,
+
+	TP_PROTO(struct tasklet_struct *t, void *func),
+
+	TP_ARGS(t, func)
+);
+
 #endif /*  _TRACE_IRQ_H */
 
 /* This part must be outside protection */
diff --git a/kernel/softirq.c b/kernel/softirq.c
index 9ab5ca399a99..fadc6bbda27b 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -822,10 +822,15 @@ static void tasklet_action_common(struct softirq_action *a,
 		if (tasklet_trylock(t)) {
 			if (!atomic_read(&t->count)) {
 				if (tasklet_clear_sched(t)) {
-					if (t->use_callback)
+					if (t->use_callback) {
+						trace_tasklet_entry(t, t->callback);
 						t->callback(t);
-					else
+						trace_tasklet_exit(t, t->callback);
+					} else {
+						trace_tasklet_entry(t, t->func);
 						t->func(t->data);
+						trace_tasklet_exit(t, t->func);
+					}
 				}
 				tasklet_unlock(t);
 				continue;
-- 
2.43.0

[PATCH 6.1] softirq: Add trace points for tasklet entry/exit

[Sumanth Gavini]

Hi All,

Just following up on my patch submitted with subject "Subject: [PATCH 6.1] softirq: Add trace points for tasklet entry/exit".

Original message: https://lore.kernel.org/all/20250812161755.609600-1-sumanth.gavini@yahoo.com/

Would you have any feedback on this change? I'd be happy to address any comments or concerns.

This patch fixes this three bugs
1. https://syzkaller.appspot.com/bug?extid=5284a86a0b0a31ab266a
2. https://syzkaller.appspot.com/bug?extid=296695c8ae3c7da3d511
3. https://syzkaller.appspot.com/bug?extid=97f2ac670e5e7a3b48e4

Thank you for your time and consideration.

Regards,
Sumanth Gavini

Re: [PATCH 6.1] softirq: Add trace points for tasklet entry/exit

[John Stultz]

On Thu, Aug 21, 2025 at 11:41 AM Sumanth Gavini
<sumanth.gavini@yahoo.com> wrote:
>
> Hi All,
>
> Just following up on my patch submitted with subject "Subject: [PATCH 6.1] softirq: Add trace points for tasklet entry/exit".
>
> Original message: https://lore.kernel.org/all/20250812161755.609600-1-sumanth.gavini@yahoo.com/
>
> Would you have any feedback on this change? I'd be happy to address any comments or concerns.
>
> This patch fixes this three bugs
> 1. https://syzkaller.appspot.com/bug?extid=5284a86a0b0a31ab266a
> 2. https://syzkaller.appspot.com/bug?extid=296695c8ae3c7da3d511
> 3. https://syzkaller.appspot.com/bug?extid=97f2ac670e5e7a3b48e4

How does a patch adding a tracepoint fix the bugs highlighted here?
It seems maybe it would help in debugging those issues, but I'm not
sure I see how it would fix them.

thanks
-john

[PATCH 6.1] softirq: Add trace points for tasklet entry/exit

[Sumanth Gavini]

>>
>> Hi All,
>>
>> Just following up on my patch submitted with subject "Subject: [PATCH 6.1] softirq: Add trace points for tasklet entry/exit".
>>
>> Original message: https://lore.kernel.org/all/20250812161755.609600-1-sumanth.gavini@yahoo.com/
>>
>> Would you have any feedback on this change? I'd be happy to address any comments or concerns.
>>
>> This patch fixes this three bugs
>> 1. https://syzkaller.appspot.com/bug?extid=5284a86a0b0a31ab266a
>> 2. https://syzkaller.appspot.com/bug?extid=296695c8ae3c7da3d511
>> 3. https://syzkaller.appspot.com/bug?extid=97f2ac670e5e7a3b48e4

> How does a patch adding a tracepoint fix the bugs highlighted here?
> It seems maybe it would help in debugging those issues, but I'm not
> sure I see how it would fix them.

This patch is related to linux 6.1/backports, the backports(https://syzkaller.appspot.com/linux-6.1/backports)
I see this patch would fix these bugs. Let me know if my understand is wrong. 

Regards,
Sumanth

Re: [PATCH 6.1] softirq: Add trace points for tasklet entry/exit

[John Stultz]

On Thu, Aug 21, 2025 at 2:02 PM Sumanth Gavini <sumanth.gavini@yahoo.com> wrote:
> >>
> >> Just following up on my patch submitted with subject "Subject: [PATCH 6.1] softirq: Add trace points for tasklet entry/exit".
> >>
> >> Original message: https://lore.kernel.org/all/20250812161755.609600-1-sumanth.gavini@yahoo.com/
> >>
> >> Would you have any feedback on this change? I'd be happy to address any comments or concerns.
> >>
> >> This patch fixes this three bugs
> >> 1. https://syzkaller.appspot.com/bug?extid=5284a86a0b0a31ab266a
> >> 2. https://syzkaller.appspot.com/bug?extid=296695c8ae3c7da3d511
> >> 3. https://syzkaller.appspot.com/bug?extid=97f2ac670e5e7a3b48e4
>
> > How does a patch adding a tracepoint fix the bugs highlighted here?
> > It seems maybe it would help in debugging those issues, but I'm not
> > sure I see how it would fix them.
>
> This patch is related to linux 6.1/backports, the backports(https://syzkaller.appspot.com/linux-6.1/backports)
> I see this patch would fix these bugs. Let me know if my understand is wrong.

But that doesn't explain why or how it fixes the bugs.  I'm not
opposed to stable taking this, but the reasoning should be clear, if
that is the motivation for including this change.
I fret there is something incidental in this patch that avoids the
problem, and that those issues may need a deeper fix rather then to
hide them with this change.

thanks
-john

Re: [PATCH 6.1] softirq: Add trace points for tasklet entry/exit

[Greg KH]

On Tue, Aug 12, 2025 at 11:17:54AM -0500, Sumanth Gavini wrote:
> commit f4bf3ca2e5cba655824b6e0893a98dfb33ed24e5 upstream.
> 
> Tasklets are supposed to finish their work quickly and should not block the
> current running process, but it is not guaranteed that they do so.
> 
> Currently softirq_entry/exit can be used to analyse the total tasklets
> execution time, but that's not helpful to track individual tasklets
> execution time. That makes it hard to identify tasklet functions, which
> take more time than expected.
> 
> Add tasklet_entry/exit trace point support to track individual tasklet
> execution.
> 
> Trivial usage example:
>    # echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_entry/enable
>    # echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_exit/enable
>    # cat /sys/kernel/debug/tracing/trace
>  # tracer: nop
>  #
>  # entries-in-buffer/entries-written: 4/4   #P:4
>  #
>  #                                _-----=> irqs-off/BH-disabled
>  #                               / _----=> need-resched
>  #                              | / _---=> hardirq/softirq
>  #                              || / _--=> preempt-depth
>  #                              ||| / _-=> migrate-disable
>  #                              |||| /     delay
>  #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
>  #              | |         |   |||||     |         |
>            <idle>-0       [003] ..s1.   314.011428: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func
>            <idle>-0       [003] ..s1.   314.011432: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func
>            <idle>-0       [003] ..s1.   314.017369: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func
>            <idle>-0       [003] ..s1.   314.017371: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func
> 
> Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
> Signed-off-by: J. Avila <elavila@google.com>
> Signed-off-by: John Stultz <jstultz@google.com>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
> Link: https://lore.kernel.org/r/20230407230526.1685443-1-jstultz@google.com
> 
> [elavila: Port to android-mainline]

This is not android-mainline, this is the normal stable tree.

And I'm with John, this makes no sense as to why you need/want these.  I
think that the syzbot report is bogus, sorry.  Please prove me wrong :)

thanks,

greg k-h

Re: [PATCH 6.1] softirq: Add trace points for tasklet entry/exit

[Thomas Gleixner]

On Fri, Aug 22 2025 at 15:07, Greg KH wrote:
> On Tue, Aug 12, 2025 at 11:17:54AM -0500, Sumanth Gavini wrote:
>
> And I'm with John, this makes no sense as to why you need/want these.  I
> think that the syzbot report is bogus, sorry.  Please prove me wrong :)

It was validated by AI (Absense of Intelligence) that adding tracepoints
makes the problem go away! So why do you want extra proof?

Thanks,

        tglx

Re: [PATCH 6.1] softirq: Add trace points for tasklet entry/exit

[Sumanth Gavini]

Hi Thomas, John,

Thanks for the feedback. Just to clarify — my intention here was only to backport the already accepted upstream changes into this branch. I do not plan to introduce any additional modifications or syzbot-only patches.

Thank you for your time and consideration.

Regards,
Sumanth Gavini

Re: [PATCH 6.1] softirq: Add trace points for tasklet entry/exit

[John Stultz]

On Tue, Sep 23, 2025 at 6:13 AM Sumanth Gavini <sumanth.gavini@yahoo.com> wrote:
>
> Hi Thomas, John,
>
> Thanks for the feedback. Just to clarify — my intention here was only to backport the already accepted upstream changes into this branch. I do not plan to introduce any additional modifications or syzbot-only patches.
>

Might be good to revisit:
https://docs.kernel.org/process/stable-kernel-rules.html

And, assuming you feel like it still might be needed, you might add
some annotation before your signed-off-by line to clarify your
motivation for submitting it, ie: how it "fix[es] a real bug that
bothers people".

Adding tracepoints definitely feels more like a feature, not a bug fix.

thanks
-john

[PATCH V2 6.1] softirq: Add trace points for tasklet entry/exit

[Sumanth Gavini]

commit f4bf3ca2e5cba655824b6e0893a98dfb33ed24e5 upstream.

Tasklets are supposed to finish their work quickly and should not block the
current running process, but it is not guaranteed that they do so.

Currently softirq_entry/exit can be used to analyse the total tasklets
execution time, but that's not helpful to track individual tasklets
execution time. That makes it hard to identify tasklet functions, which
take more time than expected.

Add tasklet_entry/exit trace point support to track individual tasklet
execution.

Trivial usage example:
   # echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_entry/enable
   # echo 1 > /sys/kernel/debug/tracing/events/irq/tasklet_exit/enable
   # cat /sys/kernel/debug/tracing/trace
 # tracer: nop
 #
 # entries-in-buffer/entries-written: 4/4   #P:4
 #
 #                                _-----=> irqs-off/BH-disabled
 #                               / _----=> need-resched
 #                              | / _---=> hardirq/softirq
 #                              || / _--=> preempt-depth
 #                              ||| / _-=> migrate-disable
 #                              |||| /     delay
 #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
 #              | |         |   |||||     |         |
           <idle>-0       [003] ..s1.   314.011428: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func
           <idle>-0       [003] ..s1.   314.011432: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func
           <idle>-0       [003] ..s1.   314.017369: tasklet_entry: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func
           <idle>-0       [003] ..s1.   314.017371: tasklet_exit: tasklet=0xffffa01ef8db2740 function=tcp_tasklet_func

Signed-off-by: Lingutla Chandrasekhar <clingutla@codeaurora.org>
Signed-off-by: J. Avila <elavila@google.com>
Signed-off-by: John Stultz <jstultz@google.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Link: https://lore.kernel.org/r/20230407230526.1685443-1-jstultz@google.com

[elavila: Port to android-mainline]
[jstultz: Rebased to upstream, cut unused trace points, added
 comments for the tracepoints, reworded commit]

The intention is to keep the stable branch in sync with upstream fixes
and improve observability without introducing new functionality.

Signed-off-by: Sumanth Gavini <sumanth.gavini@yahoo.com>

Changes in V2:
- No code changes
- Link to V1: https://lore.kernel.org/all/20250812161755.609600-1-sumanth.gavini@yahoo.com/
- Updated the comment msg before the signed-off-by
---
 include/trace/events/irq.h | 47 ++++++++++++++++++++++++++++++++++++++
 kernel/softirq.c           |  9 ++++++--
 2 files changed, 54 insertions(+), 2 deletions(-)

diff --git a/include/trace/events/irq.h b/include/trace/events/irq.h
index eeceafaaea4c..a07b4607b663 100644
--- a/include/trace/events/irq.h
+++ b/include/trace/events/irq.h
@@ -160,6 +160,53 @@ DEFINE_EVENT(softirq, softirq_raise,
 	TP_ARGS(vec_nr)
 );
 
+DECLARE_EVENT_CLASS(tasklet,
+
+	TP_PROTO(struct tasklet_struct *t, void *func),
+
+	TP_ARGS(t, func),
+
+	TP_STRUCT__entry(
+		__field(	void *,	tasklet)
+		__field(	void *,	func)
+	),
+
+	TP_fast_assign(
+		__entry->tasklet = t;
+		__entry->func = func;
+	),
+
+	TP_printk("tasklet=%ps function=%ps", __entry->tasklet, __entry->func)
+);
+
+/**
+ * tasklet_entry - called immediately before the tasklet is run
+ * @t: tasklet pointer
+ * @func: tasklet callback or function being run
+ *
+ * Used to find individual tasklet execution time
+ */
+DEFINE_EVENT(tasklet, tasklet_entry,
+
+	TP_PROTO(struct tasklet_struct *t, void *func),
+
+	TP_ARGS(t, func)
+);
+
+/**
+ * tasklet_exit - called immediately after the tasklet is run
+ * @t: tasklet pointer
+ * @func: tasklet callback or function being run
+ *
+ * Used to find individual tasklet execution time
+ */
+DEFINE_EVENT(tasklet, tasklet_exit,
+
+	TP_PROTO(struct tasklet_struct *t, void *func),
+
+	TP_ARGS(t, func)
+);
+
 #endif /*  _TRACE_IRQ_H */
 
 /* This part must be outside protection */
diff --git a/kernel/softirq.c b/kernel/softirq.c
index 9ab5ca399a99..fadc6bbda27b 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -822,10 +822,15 @@ static void tasklet_action_common(struct softirq_action *a,
 		if (tasklet_trylock(t)) {
 			if (!atomic_read(&t->count)) {
 				if (tasklet_clear_sched(t)) {
-					if (t->use_callback)
+					if (t->use_callback) {
+						trace_tasklet_entry(t, t->callback);
 						t->callback(t);
-					else
+						trace_tasklet_exit(t, t->callback);
+					} else {
+						trace_tasklet_entry(t, t->func);
 						t->func(t->data);
+						trace_tasklet_exit(t, t->func);
+					}
 				}
 				tasklet_unlock(t);
 				continue;
-- 
2.43.0

Re: [PATCH V2 6.1] softirq: Add trace points for tasklet entry/exit

[Sasha Levin]

Subject: softirq: Add trace points for tasklet entry/exit

Thanks!

Re: [PATCH V2 6.1] softirq: Add trace points for tasklet entry/exit

[Steven Rostedt]

On Tue, 11 Nov 2025 21:16:20 -0600
Sumanth Gavini <sumanth.gavini@yahoo.com> wrote:

> [elavila: Port to android-mainline]
> [jstultz: Rebased to upstream, cut unused trace points, added
>  comments for the tracepoints, reworded commit]
> 
> The intention is to keep the stable branch in sync with upstream fixes
> and improve observability without introducing new functionality.
> 
> Signed-off-by: Sumanth Gavini <sumanth.gavini@yahoo.com>

I just noticed that this is being pulled into 6.1.

I'm still not sure why this was added. Yes, it may improve observability,
but that's because it is adding new functionality. Adding a trace event, is
functionality and not a bug fix.

I'm not going to argue against this backport, but I do want to point out
that it is adding new functionality to a stable release.

-- Steve