[PATCH] hung_task: fix missing hung task detection for kthread in TASK_WAKEKILL state

[PATCH] hung_task: fix missing hung task detection for kthread in TASK_WAKEKILL state

[Yafang Shao]

We recently encountered an XFS deadlock issue, which is a known problem
resolved in the upstream kernel [0]. During the analysis of this issue, I
observed that a kernel thread in the TASK_WAKEKILL state could not be
detected as a hung task by the hung_task detector. The details are as
follows:

Using the following command, I identified nine tasks stuck in the D state:

$ ps -eLo state,comm,tid,wchan  | grep ^D
D java            4177339 xfs_buf_lock
D kworker/93:3+xf 3025535 xfs_buf_lock
D kworker/87:0+xf 3426612 xfs_extent_busy_flush
D kworker/85:0+xf 3479378 xfs_buf_lock
D kworker/91:1+xf 3584478 xfs_buf_lock
D kworker/80:3+xf 3655680 xfs_buf_lock
D kworker/89:0+xf 3671691 xfs_buf_lock
D kworker/84:1+xf 3708397 xfs_buf_lock
D kworker/81:1+xf 4005763 xfs_buf_lock

However, the hung_task detector only reported eight of these tasks:

[3108840.650652] INFO: task java:4177339 blocked for more than 247779 seconds.
[3108840.654197] INFO: task kworker/93:3:3025535 blocked for more than 248427 seconds.
[3108840.657711] INFO: task kworker/85:0:3479378 blocked for more than 247836 seconds.
[3108840.661483] INFO: task kworker/91:1:3584478 blocked for more than 249638 seconds.
[3108840.664871] INFO: task kworker/80:3:3655680 blocked for more than 249638 seconds.
[3108840.668495] INFO: task kworker/89:0:3671691 blocked for more than 249047 seconds.
[3108840.672418] INFO: task kworker/84:1:3708397 blocked for more than 247836 seconds.
[3108840.676175] INFO: task kworker/81:1:4005763 blocked for more than 247836 seconds.

Task 3426612, although in the D state, was not reported as a hung task.

I confirmed that task 3426612 remained in the D (disk sleep) state and
experienced no context switches over a long period:

$ cat /proc/3426612/status | grep -E "State:|ctxt_switches:";   \
  sleep 60; echo "----"; \
  cat /proc/3426612/status | grep -E "State:|ctxt_switches:"
State:  D (disk sleep)
voluntary_ctxt_switches:        7516
nonvoluntary_ctxt_switches:     0
----
State:  D (disk sleep)
voluntary_ctxt_switches:        7516
nonvoluntary_ctxt_switches:     0

The system's hung_task detector settings were configured as follows:

  kernel.hung_task_timeout_secs = 28
  kernel.hung_task_warnings = -1

The issue lies in the handling of task state in the XFS code. Specifically,
the thread in question (3426612) was set to the TASK_KILLABLE state in
xfs_extent_busy_flush():

  xfs_extent_busy_flush
    prepare_to_wait(&pag->pagb_wait, &wait, TASK_KILLABLE);

When a task is in the TASK_WAKEKILL state (a subset of TASK_KILLABLE), the
hung_task detector ignores it, as it assumes such tasks can be terminated.
However, in this case, the kernel thread cannot be killed, meaning it
effectively becomes a hung task.

To address this issue, the hung_task detector should report the kthreads in
the TASK_WAKEKILL state.

Link: https://lore.kernel.org/linux-xfs/20230620002021.1038067-5-david@fromorbit.com/ [0]
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
Cc: Dave Chinner <david@fromorbit.com>
---
 kernel/hung_task.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index c18717189f32..ed63fd84ce2e 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -220,8 +220,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 		 */
 		state = READ_ONCE(t->__state);
 		if ((state & TASK_UNINTERRUPTIBLE) &&
+		    (t->flags & PF_KTHREAD ||
 		    !(state & TASK_WAKEKILL) &&
-		    !(state & TASK_NOLOAD))
+		    !(state & TASK_NOLOAD)))
 			check_hung_task(t, timeout);
 	}
  unlock:
-- 
2.43.5

Re: [PATCH] hung_task: fix missing hung task detection for kthread in TASK_WAKEKILL state

[Yafang Shao]

On Mon, Dec 23, 2024 at 5:37 PM Yafang Shao <laoar.shao@gmail.com> wrote:
>
> We recently encountered an XFS deadlock issue, which is a known problem
> resolved in the upstream kernel [0]. During the analysis of this issue, I
> observed that a kernel thread in the TASK_WAKEKILL state could not be
> detected as a hung task by the hung_task detector. The details are as
> follows:
>
> Using the following command, I identified nine tasks stuck in the D state:
>
> $ ps -eLo state,comm,tid,wchan  | grep ^D
> D java            4177339 xfs_buf_lock
> D kworker/93:3+xf 3025535 xfs_buf_lock
> D kworker/87:0+xf 3426612 xfs_extent_busy_flush
> D kworker/85:0+xf 3479378 xfs_buf_lock
> D kworker/91:1+xf 3584478 xfs_buf_lock
> D kworker/80:3+xf 3655680 xfs_buf_lock
> D kworker/89:0+xf 3671691 xfs_buf_lock
> D kworker/84:1+xf 3708397 xfs_buf_lock
> D kworker/81:1+xf 4005763 xfs_buf_lock
>
> However, the hung_task detector only reported eight of these tasks:
>
> [3108840.650652] INFO: task java:4177339 blocked for more than 247779 seconds.
> [3108840.654197] INFO: task kworker/93:3:3025535 blocked for more than 248427 seconds.
> [3108840.657711] INFO: task kworker/85:0:3479378 blocked for more than 247836 seconds.
> [3108840.661483] INFO: task kworker/91:1:3584478 blocked for more than 249638 seconds.
> [3108840.664871] INFO: task kworker/80:3:3655680 blocked for more than 249638 seconds.
> [3108840.668495] INFO: task kworker/89:0:3671691 blocked for more than 249047 seconds.
> [3108840.672418] INFO: task kworker/84:1:3708397 blocked for more than 247836 seconds.
> [3108840.676175] INFO: task kworker/81:1:4005763 blocked for more than 247836 seconds.
>
> Task 3426612, although in the D state, was not reported as a hung task.
>
> I confirmed that task 3426612 remained in the D (disk sleep) state and
> experienced no context switches over a long period:
>
> $ cat /proc/3426612/status | grep -E "State:|ctxt_switches:";   \
>   sleep 60; echo "----"; \
>   cat /proc/3426612/status | grep -E "State:|ctxt_switches:"
> State:  D (disk sleep)
> voluntary_ctxt_switches:        7516
> nonvoluntary_ctxt_switches:     0
> ----
> State:  D (disk sleep)
> voluntary_ctxt_switches:        7516
> nonvoluntary_ctxt_switches:     0
>
> The system's hung_task detector settings were configured as follows:
>
>   kernel.hung_task_timeout_secs = 28
>   kernel.hung_task_warnings = -1
>
> The issue lies in the handling of task state in the XFS code. Specifically,
> the thread in question (3426612) was set to the TASK_KILLABLE state in
> xfs_extent_busy_flush():
>
>   xfs_extent_busy_flush
>     prepare_to_wait(&pag->pagb_wait, &wait, TASK_KILLABLE);
>
> When a task is in the TASK_WAKEKILL state (a subset of TASK_KILLABLE), the
> hung_task detector ignores it, as it assumes such tasks can be terminated.
> However, in this case, the kernel thread cannot be killed, meaning it
> effectively becomes a hung task.
>
> To address this issue, the hung_task detector should report the kthreads in
> the TASK_WAKEKILL state.
>
> Link: https://lore.kernel.org/linux-xfs/20230620002021.1038067-5-david@fromorbit.com/ [0]
> Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
> Cc: Dave Chinner <david@fromorbit.com>
> ---
>  kernel/hung_task.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/hung_task.c b/kernel/hung_task.c
> index c18717189f32..ed63fd84ce2e 100644
> --- a/kernel/hung_task.c
> +++ b/kernel/hung_task.c
> @@ -220,8 +220,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
>                  */
>                 state = READ_ONCE(t->__state);
>                 if ((state & TASK_UNINTERRUPTIBLE) &&
> +                   (t->flags & PF_KTHREAD ||
>                     !(state & TASK_WAKEKILL) &&
> -                   !(state & TASK_NOLOAD))
> +                   !(state & TASK_NOLOAD)))
>                         check_hung_task(t, timeout);
>         }
>   unlock:
> --
> 2.43.5
>

Please disregard this. There may be multiple hung tasks in the TASK_IDLE state.
I will send a new one.

--
Regards
Yafang