[PATCH] sched: Dump stacktrace when some task failed to stop

[PATCH] sched: Dump stacktrace when some task failed to stop

[Schspa Shi]

Some processes may not be stopped, such as workqueue workers.

Add this dump_stack to easily figure out which process caused the problems.

Before this change:

<4>[  517.545889] ------------[ cut here ]------------
<4>[  517.545901] Dying CPU not properly vacated!
<4>[  517.546366] ---[ end trace 0000000000000002 ]---
<4>[  518.546831] CPU6 enqueued tasks (2 total):
<4>[  518.546855] 	pid: 49, name: migration/6
<4>[  518.547010] 	pid: 6231, name: kworker/6:0

Signed-off-by: Schspa Shi <schspa@gmail.com>
---
 kernel/sched/core.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 53596842f0d8..de6b5320e4d7 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -9416,7 +9416,14 @@ static void dump_rq_tasks(struct rq *rq, const char *loglvl)
 			continue;
 
 		printk("%s\tpid: %d, name: %s\n", loglvl, p->pid, p->comm);
+		if (p != current)
+			sched_show_task(p);
 	}
+	/*
+	 * We need the stack of each cpu to analyze who is blocking the
+	 * current cpu scheduling
+	 */
+	trigger_all_cpu_backtrace();
 }
 
 int sched_cpu_dying(unsigned int cpu)
-- 
2.24.3 (Apple Git-128)

Re: [PATCH] sched: Dump stacktrace when some task failed to stop

[Valentin Schneider]

On 20/05/22 00:11, Schspa Shi wrote:
> Some processes may not be stopped, such as workqueue workers.
>
> Add this dump_stack to easily figure out which process caused the problems.
>
> Before this change:
>
> <4>[  517.545889] ------------[ cut here ]------------
> <4>[  517.545901] Dying CPU not properly vacated!
> <4>[  517.546366] ---[ end trace 0000000000000002 ]---
> <4>[  518.546831] CPU6 enqueued tasks (2 total):
> <4>[  518.546855] 	pid: 49, name: migration/6
> <4>[  518.547010] 	pid: 6231, name: kworker/6:0
>
> Signed-off-by: Schspa Shi <schspa@gmail.com>
> ---
>  kernel/sched/core.c | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 53596842f0d8..de6b5320e4d7 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -9416,7 +9416,14 @@ static void dump_rq_tasks(struct rq *rq, const char *loglvl)
>  			continue;
>  
>  		printk("%s\tpid: %d, name: %s\n", loglvl, p->pid, p->comm);
> +		if (p != current)
> +			sched_show_task(p);
>  	}
> +	/*
> +	 * We need the stack of each cpu to analyze who is blocking the
> +	 * current cpu scheduling
> +	 */
> +	trigger_all_cpu_backtrace();
>  }
>

Given you get a WARN() when this happens, can't you get the same thing with

  panic_on_warn=1
  panic_print=PANIC_PRINT_ALL_CPU_BT

?

>  int sched_cpu_dying(unsigned int cpu)
> -- 
> 2.24.3 (Apple Git-128)

Re: [PATCH] sched: Dump stacktrace when some task failed to stop

[Schspa Shi]

Valentin Schneider <vschneid@redhat.com> writes:

> On 20/05/22 00:11, Schspa Shi wrote:
>> Some processes may not be stopped, such as workqueue workers.
>>
>> Add this dump_stack to easily figure out which process caused the problems.
>>
>> Before this change:
>>
>> <4>[  517.545889] ------------[ cut here ]------------
>> <4>[  517.545901] Dying CPU not properly vacated!
>> <4>[  517.546366] ---[ end trace 0000000000000002 ]---
>> <4>[  518.546831] CPU6 enqueued tasks (2 total):
>> <4>[  518.546855]    pid: 49, name: migration/6
>> <4>[  518.547010]    pid: 6231, name: kworker/6:0
>>
>> Signed-off-by: Schspa Shi <schspa@gmail.com>
>> ---
>>  kernel/sched/core.c | 7 +++++++
>>  1 file changed, 7 insertions(+)
>>
>> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
>> index 53596842f0d8..de6b5320e4d7 100644
>> --- a/kernel/sched/core.c
>> +++ b/kernel/sched/core.c
>> @@ -9416,7 +9416,14 @@ static void dump_rq_tasks(struct rq *rq, const char *loglvl)
>>                      continue;
>>
>>              printk("%s\tpid: %d, name: %s\n", loglvl, p->pid, p->comm);
>> +            if (p != current)
>> +                    sched_show_task(p);
>>      }
>> +    /*
>> +     * We need the stack of each cpu to analyze who is blocking the
>> +     * current cpu scheduling
>> +     */
>> +    trigger_all_cpu_backtrace();
>>  }
>>
>
> Given you get a WARN() when this happens, can't you get the same thing with
>
>   panic_on_warn=1
>   panic_print=PANIC_PRINT_ALL_CPU_BT
>
> ?

Yes, I can get the information with these two parameters set.

But this is generally a difficult problem to reproduce, so wanted
to print it here so that useful information can be obtained without
turning on these switches.

>
>>  int sched_cpu_dying(unsigned int cpu)
>> --
>> 2.24.3 (Apple Git-128)

--
BRs

Schspa