[PATCH] workqueue: show the latest function name in /proc/PID/{comm,stat,status}

[PATCH] workqueue: show the latest function name in /proc/PID/{comm,stat,status}

[Muchun Song]

Workqueues, such as system_unbound_wq, are shared across the system,
making it difficult to determine which tasks are being executed by
each worker. This patch improves clarity by displaying the latest
function name associated with each workqueue worker in the
/proc/PID/{comm,stat,status}. For example:

Before:
  # ps 64 67 68
  PID TTY STAT TIME COMMAND
  64  ?   I    0:00 [kworker/u34:0-events_unbound]
  67  ?   I    0:00 [kworker/u33:1-events_unbound]
  68  ?   I    0:00 [kworker/u33:2-events_unbound]

After:
  # ps 64 67 68
  PID TTY STAT TIME COMMAND
  64  ?   I    0:00 [kworker/u34:0-events_unbound:flush_memcg_stats_dwork]
  67  ?   I    0:00 [kworker/u33:1-events_unbound:flush_to_ldisc]
  68  ?   I    0:00 [kworker/u33:2-events_unbound:idle_cull_fn]

This change provides a clearer view of the tasks being performed by
each worker, enhancing system monitoring and debugging.

Signed-off-by: Muchun Song <songmuchun@bytedance.com>
---
 kernel/workqueue.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 3bef0754cf73..6d4ca373b980 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -6475,9 +6475,12 @@ void wq_worker_comm(char *buf, size_t size, struct task_struct *task)
 			 * current, prepend '+', otherwise '-'.
 			 */
 			if (worker->desc[0] != '\0') {
-				if (worker->current_work)
-					scnprintf(buf + off, size - off, "+%s",
-						  worker->desc);
+				if (worker->current_func)
+					scnprintf(buf + off, size - off, "+%s:%ps",
+						  worker->desc, worker->current_func);
+				else if (worker->last_func)
+					scnprintf(buf + off, size - off, "-%s:%ps",
+						  worker->desc, worker->last_func);
 				else
 					scnprintf(buf + off, size - off, "-%s",
 						  worker->desc);
-- 
2.20.1

Re: [PATCH] workqueue: show the latest function name in /proc/PID/{comm,stat,status}

[Tejun Heo]

Hello, Muchun.

On Thu, May 15, 2025 at 12:05:23PM +0800, Muchun Song wrote:
> Workqueues, such as system_unbound_wq, are shared across the system,
> making it difficult to determine which tasks are being executed by
> each worker. This patch improves clarity by displaying the latest
> function name associated with each workqueue worker in the
> /proc/PID/{comm,stat,status}. For example:
> 
> Before:
>   # ps 64 67 68
>   PID TTY STAT TIME COMMAND
>   64  ?   I    0:00 [kworker/u34:0-events_unbound]
>   67  ?   I    0:00 [kworker/u33:1-events_unbound]
>   68  ?   I    0:00 [kworker/u33:2-events_unbound]
> 
> After:
>   # ps 64 67 68
>   PID TTY STAT TIME COMMAND
>   64  ?   I    0:00 [kworker/u34:0-events_unbound:flush_memcg_stats_dwork]
>   67  ?   I    0:00 [kworker/u33:1-events_unbound:flush_to_ldisc]
>   68  ?   I    0:00 [kworker/u33:2-events_unbound:idle_cull_fn]
> 
> This change provides a clearer view of the tasks being performed by
> each worker, enhancing system monitoring and debugging.

One worry I have about this is that these will likely be stale most of the
time as kworkers are just sitting and waiting, and there's no connection
between what a kworker ran before and what it's going to run next, so I'm
unsure how useful the extra information is.

Thanks.

-- 
tejun

Re: [PATCH] workqueue: show the latest function name in /proc/PID/{comm,stat,status}

[Muchun Song]



> On May 15, 2025, at 23:53, Tejun Heo <tj@kernel.org> wrote:
> 
> Hello, Muchun.
> 
> On Thu, May 15, 2025 at 12:05:23PM +0800, Muchun Song wrote:
>> Workqueues, such as system_unbound_wq, are shared across the system,
>> making it difficult to determine which tasks are being executed by
>> each worker. This patch improves clarity by displaying the latest
>> function name associated with each workqueue worker in the
>> /proc/PID/{comm,stat,status}. For example:
>> 
>> Before:
>>  # ps 64 67 68
>>  PID TTY STAT TIME COMMAND
>>  64  ?   I    0:00 [kworker/u34:0-events_unbound]
>>  67  ?   I    0:00 [kworker/u33:1-events_unbound]
>>  68  ?   I    0:00 [kworker/u33:2-events_unbound]
>> 
>> After:
>>  # ps 64 67 68
>>  PID TTY STAT TIME COMMAND
>>  64  ?   I    0:00 [kworker/u34:0-events_unbound:flush_memcg_stats_dwork]
>>  67  ?   I    0:00 [kworker/u33:1-events_unbound:flush_to_ldisc]
>>  68  ?   I    0:00 [kworker/u33:2-events_unbound:idle_cull_fn]
>> 
>> This change provides a clearer view of the tasks being performed by
>> each worker, enhancing system monitoring and debugging.
> 
> One worry I have about this is that these will likely be stale most of the
> time as kworkers are just sitting and waiting, and there's no connection
> between what a kworker ran before and what it's going to run next, so I'm
> unsure how useful the extra information is.

Hi Tejun,

Monitoring tools like atop can indeed record comm of processes. When we
encounter issues such as high CPU usage, these tools can help us identify
the problem. For instance, if kworkers are consuming most of the CPU, we
can use this information to pinpoint which specific function is using
the most CPU.

Another use case is when we use the isolcpus= command line option
to isolate CPUs, we want to make sure that no kworker threads run on
those CPUs. But sometimes, kworkers might still get scheduled there, causing latency
issues. By using this information, we can figure out which module's
function ran on the CPU before and then dig into the code to see how to
stop it from happening.


Muchun,
Thanks

> 
> Thanks.
> 
> -- 
> tejun

Re: [PATCH] workqueue: show the latest function name in /proc/PID/{comm,stat,status}

[Tejun Heo]

Hello,

On Fri, May 16, 2025 at 12:44:19PM +0800, Muchun Song wrote:
...
> Monitoring tools like atop can indeed record comm of processes. When we
> encounter issues such as high CPU usage, these tools can help us identify
> the problem. For instance, if kworkers are consuming most of the CPU, we
> can use this information to pinpoint which specific function is using
> the most CPU.

You can get more detailed information with `cat /proc/KWORKER_PID/stack`.
The problem is that last_func can be stale most of the time and can be more
misleading than useful.

> Another use case is when we use the isolcpus= command line option
> to isolate CPUs, we want to make sure that no kworker threads run on
> those CPUs. But sometimes, kworkers might still get scheduled there, causing latency
> issues. By using this information, we can figure out which module's
> function ran on the CPU before and then dig into the code to see how to
> stop it from happening.

Can you explain why `cat /proc/KWORKER_PID/stack` can't be used for that? If
the worker runtime is too short to capture, you can easily use tracepoints
or bpftrace too. What are the benefits of showing last_func?

Thanks.

-- 
tejun