[PATCH] hung_task: Skip hung task detection during core dump operations

[PATCH] hung_task: Skip hung task detection during core dump operations

[Nanji Parmar (he/him)]

[-- Attachment #1.1: Type: text/plain, Size: 1981 bytes --]

Hi,

This patch fixes false positive hung task warnings during core dump
operations for processes with large memory footprints.

During testing with processes having GBs or >1TB memory, core dump
generation
takes many minutes, causing hung task detector to incorrectly flag threads
as hung. The fix checks for both PF_DUMPCORE and PF_POSTCOREDUMP flags
before reporting tasks as hung.

Tested on the systems with large memory processes.

Best regards,
Nanji

---
From 45460c6882b602669b25a57f3a2f7ea8a8ea0f84 Mon Sep 17 00:00:00 2001
From: Nanji Parmar <nparmar@purestorage.com>
Date: Wed, 13 Aug 2025 12:14:35 -0600
Subject: [PATCH] hung_task: Exclude core dump tasks from hung task detection

Tasks involved in core dump operations can legitimately block for
extended periods, especially for large memory processes. The hung
task detector should skip tasks with PF_DUMPCORE (main dumping
thread) or PF_POSTCOREDUMP (other threads in the group) flags to
avoid false positive warnings.

This prevents incorrect hung task reports during legitimate core
dump generation that can take xx minutes for large processes.

Signed-off-by: Nanji Parmar <nparmar@purestorage.com>
---
 kernel/hung_task.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 8708a1205f82..0fc3352d0f0e 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -217,6 +217,13 @@ static void check_hung_task(struct task_struct *t,
unsigned long timeout)
         */
        sysctl_hung_task_detect_count++;

+       /* Skip hung task detection for tasks involved in core dump
operations */
+       if (t->flags & (PF_DUMPCORE | PF_POSTCOREDUMP)) {
+               pr_info("Skipping hung task check for coredump-related task
%s:%d (blocked %ld seconds)\n",
+                       t->comm, t->pid, (jiffies - t->last_switch_time) /
HZ);
+               return;
+       }
+
        trace_sched_process_hang(t);

        if (sysctl_hung_task_panic) {
--
2.50.1

[-- Attachment #1.2: Type: text/html, Size: 2360 bytes --]

[-- Attachment #2: hung_task_fix.patch --]
[-- Type: application/octet-stream, Size: 1434 bytes --]

From 45460c6882b602669b25a57f3a2f7ea8a8ea0f84 Mon Sep 17 00:00:00 2001
From: Nanji Parmar <nparmar@purestorage.com>
Date: Wed, 13 Aug 2025 12:14:35 -0600
Subject: [PATCH] hung_task: Exclude core dump tasks from hung task detection

Tasks involved in core dump operations can legitimately block for
extended periods, especially for large memory processes. The hung
task detector should skip tasks with PF_DUMPCORE (main dumping
thread) or PF_POSTCOREDUMP (other threads in the group) flags to
avoid false positive warnings.

This prevents incorrect hung task reports during legitimate core
dump generation that can take xx minutes for large processes.

Signed-off-by: Nanji Parmar <nparmar@purestorage.com>
---
 kernel/hung_task.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 8708a1205f82..0fc3352d0f0e 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -217,6 +217,13 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)
 	 */
 	sysctl_hung_task_detect_count++;
 
+	/* Skip hung task detection for tasks involved in core dump operations */
+	if (t->flags & (PF_DUMPCORE | PF_POSTCOREDUMP)) {
+		pr_info("Skipping hung task check for coredump-related task %s:%d (blocked %ld seconds)\n",
+			t->comm, t->pid, (jiffies - t->last_switch_time) / HZ);
+		return;
+	}
+
 	trace_sched_process_hang(t);
 
 	if (sysctl_hung_task_panic) {
-- 
2.50.1

Re: [PATCH] hung_task: Skip hung task detection during core dump operations

[Andrew Morton]

On Wed, 13 Aug 2025 11:30:36 -0700 "Nanji Parmar (he/him)" <nparmar@purestorage.com> wrote:

> Tasks involved in core dump operations can legitimately block for
> extended periods, especially for large memory processes. The hung
> task detector should skip tasks with PF_DUMPCORE (main dumping
> thread) or PF_POSTCOREDUMP (other threads in the group) flags to
> avoid false positive warnings.
> 
> This prevents incorrect hung task reports during legitimate core
> dump generation that can take xx minutes for large processes.

It isn't pleasing to be putting coredump special cases into the core of
the hung-task detector.  Perhaps the hung task detector should get an
equivalent to touch_softlockup_watchdog().  I'm surprised it doesn't
already have such a thing.  Maybe it does and I've forgotten where it is.

Please provide a full description of the problem, mainly the relevant
dmesg output.  Please always provide this full description when
addressing kernel issues, thanks.

Re: [PATCH] hung_task: Skip hung task detection during core dump operations

[Lance Yang]

Hi Nanji,

Thanks for your patch!

On 2025/8/14 06:01, Andrew Morton wrote:
> On Wed, 13 Aug 2025 11:30:36 -0700 "Nanji Parmar (he/him)" <nparmar@purestorage.com> wrote:
> 
>> Tasks involved in core dump operations can legitimately block for
>> extended periods, especially for large memory processes. The hung
>> task detector should skip tasks with PF_DUMPCORE (main dumping
>> thread) or PF_POSTCOREDUMP (other threads in the group) flags to
>> avoid false positive warnings.
>>
>> This prevents incorrect hung task reports during legitimate core
>> dump generation that can take xx minutes for large processes.
> 
> It isn't pleasing to be putting coredump special cases into the core of
> the hung-task detector.  Perhaps the hung task detector should get an

Yeah, adding a special case for coredumps is not a good design ;)

> equivalent to touch_softlockup_watchdog().  I'm surprised it doesn't
> already have such a thing.  Maybe it does and I've forgotten where it is.
> 
> Please provide a full description of the problem, mainly the relevant
> dmesg output.  Please always provide this full description when
> addressing kernel issues, thanks.

Interestingly, I wasn't able to reproduce the hung task warning on my
machine with a SSD, even when generating a 100 GiB coredump. The process
switches between R and D states so fast that it never hits the timeout,
even with hung_task_timeout_secs set as low as 5s ;)

So it seems this isn't a general problem for all coredumps. It look like
it only happens on systems with slow I/O, which can cause a process to
stay in a D-state for a long time.

Anyway, any task *actually* blocked on I/O for that long should be flagged;
that is the hung task detector's job, IMHO.

Thanks,
Lance

Re: [PATCH] hung_task: Skip hung task detection during core dump operations

[Lance Yang]

On 2025/8/14 11:31, Nanji Parmar (he/him) wrote:
> Hi Lance, Andrew,
> 
> Thanks for looking into this.
> After checking further, we found that the following patch fixed that 
> issue. Thank you once again.

Ah, I see. That's why I couldn't reproduce it on 6.16 kernel — the
fix was already there ;)

Thanks for digging this up!
Lance


> 
> commit b8e753128ed074fcb48e9ceded940752f6b1c19f
> Author: Paul E. McKenney <paulmck@kernel.org <mailto:paulmck@kernel.org>>
> Date:   Wed Jul 24 17:51:52 2024
> 
>      exit: Sleep at TASK_IDLE when waiting for application core dump
> 
>      Currently, the coredump_task_exit() function sets the task state
>      to TASK_UNINTERRUPTIBLE|TASK_FREEZABLE, which usually works well.
>      But a combination of large memory and slow (and/or highly contended)
>      mass storage can cause application core dumps to take more than
>      two minutes, which can cause check_hung_task(), which is invoked by
>      check_hung_uninterruptible_tasks(), to produce task-blocked splats.
>      There does not seem to be any reasonable benefit to getting these 
> splats.
> 
>      Furthermore, as Oleg Nesterov points out, TASK_UNINTERRUPTIBLE could
>      be misleading because the task sleeping in coredump_task_exit() really
>      is killable, albeit indirectly.  See the check of signal->core_state
>      in prepare_signal() and the check of fatal_signal_pending()
>      in dump_interrupted(), which bypass the normal unkillability of
>      TASK_UNINTERRUPTIBLE, resulting in coredump_finish() invoking
>      wake_up_process() on any threads sleeping in coredump_task_exit().
> 
>      Therefore, change that TASK_UNINTERRUPTIBLE to TASK_IDLE.
> 
>      Reported-by: Anhad Jai Singh <ffledgling@meta.com 
> <mailto:ffledgling@meta.com>>
>      Signed-off-by: Paul E. McKenney <paulmck@kernel.org 
> <mailto:paulmck@kernel.org>>
>      Acked-by: Oleg Nesterov <oleg@redhat.com <mailto:oleg@redhat.com>>
>      Cc: Jens Axboe <axboe@kernel.dk <mailto:axboe@kernel.dk>>
>      Cc: Christian Brauner <brauner@kernel.org <mailto:brauner@kernel.org>>
>      Cc: Andrew Morton <akpm@linux-foundation.org <mailto:akpm@linux- 
> foundation.org>>
>      Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org 
> <mailto:willy@infradead.org>>
>      Cc: Chris Mason <clm@fb.com <mailto:clm@fb.com>>
>      Cc: Rik van Riel <riel@surriel.com <mailto:riel@surriel.com>>
> 
> diff --git a/kernel/exit.c b/kernel/exit.c
> index 7430852a8571..0d62a53605df 100644
> --- a/kernel/exit.c
> +++ b/kernel/exit.c
> @@ -428,7 +428,7 @@ static void coredump_task_exit(struct task_struct *tsk)
>                          complete(&core_state->startup);
> 
>                  for (;;) {
> -                       set_current_state(TASK_UNINTERRUPTIBLE| 
> TASK_FREEZABLE);
> +                       set_current_state(TASK_IDLE|TASK_FREEZABLE);
>                          if (!self.task) /* see coredump_finish() */
>                                  break;
>                          schedule();
> 
> Thanks,
> Nanji
> 
> On Wed, Aug 13, 2025 at 8:12 PM Lance Yang <lance.yang@linux.dev 
> <mailto:lance.yang@linux.dev>> wrote:
> 
>     Hi Nanji,
> 
>     Thanks for your patch!
> 
>     On 2025/8/14 06:01, Andrew Morton wrote:
>      > On Wed, 13 Aug 2025 11:30:36 -0700 "Nanji Parmar (he/him)"
>     <nparmar@purestorage.com <mailto:nparmar@purestorage.com>> wrote:
>      >
>      >> Tasks involved in core dump operations can legitimately block for
>      >> extended periods, especially for large memory processes. The hung
>      >> task detector should skip tasks with PF_DUMPCORE (main dumping
>      >> thread) or PF_POSTCOREDUMP (other threads in the group) flags to
>      >> avoid false positive warnings.
>      >>
>      >> This prevents incorrect hung task reports during legitimate core
>      >> dump generation that can take xx minutes for large processes.
>      >
>      > It isn't pleasing to be putting coredump special cases into the
>     core of
>      > the hung-task detector.  Perhaps the hung task detector should get an
> 
>     Yeah, adding a special case for coredumps is not a good design ;)
> 
>      > equivalent to touch_softlockup_watchdog().  I'm surprised it doesn't
>      > already have such a thing.  Maybe it does and I've forgotten
>     where it is.
>      >
>      > Please provide a full description of the problem, mainly the relevant
>      > dmesg output.  Please always provide this full description when
>      > addressing kernel issues, thanks.
> 
>     Interestingly, I wasn't able to reproduce the hung task warning on my
>     machine with a SSD, even when generating a 100 GiB coredump. The process
>     switches between R and D states so fast that it never hits the timeout,
>     even with hung_task_timeout_secs set as low as 5s ;)
> 
>     So it seems this isn't a general problem for all coredumps. It look like
>     it only happens on systems with slow I/O, which can cause a process to
>     stay in a D-state for a long time.
> 
>     Anyway, any task *actually* blocked on I/O for that long should be
>     flagged;
>     that is the hung task detector's job, IMHO.
> 
>     Thanks,
>     Lance
> 
> 

Re: [PATCH] hung_task: Skip hung task detection during core dump operations

[Masami Hiramatsu]

On Wed, 13 Aug 2025 15:01:55 -0700
Andrew Morton <akpm@linux-foundation.org> wrote:

> On Wed, 13 Aug 2025 11:30:36 -0700 "Nanji Parmar (he/him)" <nparmar@purestorage.com> wrote:
> 
> > Tasks involved in core dump operations can legitimately block for
> > extended periods, especially for large memory processes. The hung
> > task detector should skip tasks with PF_DUMPCORE (main dumping
> > thread) or PF_POSTCOREDUMP (other threads in the group) flags to
> > avoid false positive warnings.
> > 
> > This prevents incorrect hung task reports during legitimate core
> > dump generation that can take xx minutes for large processes.
> 
> It isn't pleasing to be putting coredump special cases into the core of
> the hung-task detector.  Perhaps the hung task detector should get an
> equivalent to touch_softlockup_watchdog().  I'm surprised it doesn't
> already have such a thing.  Maybe it does and I've forgotten where it is.

Hmm, maybe we can increase nvcsw/nivcsw to reset the hung task checker.
But usually this means the task does context switch while core-dump.

> 
> Please provide a full description of the problem, mainly the relevant
> dmesg output.  Please always provide this full description when
> addressing kernel issues, thanks.

+1, dmesg will show where (in kernel) we hit the hung_task during core dump.

Thanks,

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>