[PATCH 0/2] hung_task: Provide runtime reset interface for hung task detector

[PATCH 0/2] hung_task: Provide runtime reset interface for hung task detector

[Aaron Tomlin]

Introduce a write-only sysfs attribute,
/sys/kernel/hung_task_detect_count_reset, to reset the total count of
detected hung tasks at runtime.

The attribute requires writing the value "1" to trigger the reset and returns
-EINVAL for any other input, ensuring robustness.

This addition is primarily justified by the need for enhanced
administrative control and improved diagnostics workflow in a running
production environment. It addresses a key limitation of the existing
mechanism: the inability to clear persistent state without resorting to a
full system reboot. The sysfs interface provides a non-disruptive, runtime
method to manage the diagnostic state.

After a system administrator investigates a potential hang and corrects the issue,
it is now possible to clear the history back to zero. This provides a clean
slate for subsequent monitoring, ensuring that any new recurrence of a hung
task is immediately reflected by a counter value greater than zero,
streamlining the post-mortem diagnostic phase.


Aaron Tomlin (2):
  hung_task: Consolidate hung task warning into an atomic log block
  hung_task: Provide runtime reset interface for hung task detector

 .../sysfs-kernel-hung_task_detect_count_reset |  8 +++
 kernel/hung_task.c                            | 68 ++++++++++++++++---
 2 files changed, 66 insertions(+), 10 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-hung_task_detect_count_reset

-- 
2.51.0

[PATCH 1/2] hung_task: Consolidate hung task warning into an atomic log block

[Aaron Tomlin]

Consolidate the multi-line console output in check_hung_task() into a new
helper function, hung_task_diagnostics().

This patch ensures the entire diagnostic block (task info, kernel
version, and sysctl advice) is logged to the ring buffer via a single
pr_err() call. This is critical in a concurrent environment to prevent
message lines from interleaving with other CPU activity, thus
maintaining contextual integrity of the warning message.

Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 kernel/hung_task.c | 37 +++++++++++++++++++++++++++----------
 1 file changed, 27 insertions(+), 10 deletions(-)

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index d2254c91450b..d5109a0994c5 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -223,6 +223,32 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti
 }
 #endif
 
+/**
+ * hung_task_diagnostics - Print structured diagnostic info for a hung task.
+ * @t: The struct task_struct of the detected hung task.
+ *
+ * This function consolidates the printing of core diagnostic information
+ * for a task found to be blocked. This approach ensures atomic logging
+ * of the multi-line message block, preventing interleaving by other
+ * console activity, thus maintaining contextual clarity.
+ */
+static void hung_task_diagnostics(struct task_struct *t)
+{
+	unsigned long blocked_secs = (jiffies - t->last_switch_time) / HZ;
+
+	pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n"
+	       "      %s %s %.*s\n"
+	       "%s\n"
+	       "\"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\""
+	       " disables this message.\n",
+	       t->comm, t->pid, blocked_secs,
+	       print_tainted(), init_utsname()->release,
+	       (int)strcspn(init_utsname()->version, " "),
+	       init_utsname()->version,
+	       (t->flags & PF_POSTCOREDUMP) ?
+	       "      Blocked by coredump." : "");
+}
+
 static void check_hung_task(struct task_struct *t, unsigned long timeout,
 		unsigned long prev_detect_count)
 {
@@ -252,16 +278,7 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout,
 	if (sysctl_hung_task_warnings || hung_task_call_panic) {
 		if (sysctl_hung_task_warnings > 0)
 			sysctl_hung_task_warnings--;
-		pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n",
-		       t->comm, t->pid, (jiffies - t->last_switch_time) / HZ);
-		pr_err("      %s %s %.*s\n",
-			print_tainted(), init_utsname()->release,
-			(int)strcspn(init_utsname()->version, " "),
-			init_utsname()->version);
-		if (t->flags & PF_POSTCOREDUMP)
-			pr_err("      Blocked by coredump.\n");
-		pr_err("\"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\""
-			" disables this message.\n");
+		hung_task_diagnostics(t);
 		sched_show_task(t);
 		debug_show_blocker(t, timeout);
 
-- 
2.51.0

Re: [PATCH 1/2] hung_task: Consolidate hung task warning into an atomic log block

[Lance Yang]

On 2025/12/9 12:12, Aaron Tomlin wrote:
> Consolidate the multi-line console output in check_hung_task() into a new
> helper function, hung_task_diagnostics().
> 
> This patch ensures the entire diagnostic block (task info, kernel
> version, and sysctl advice) is logged to the ring buffer via a single
> pr_err() call. This is critical in a concurrent environment to prevent
> message lines from interleaving with other CPU activity, thus
> maintaining contextual integrity of the warning message.
> 
> Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
> ---
>   kernel/hung_task.c | 37 +++++++++++++++++++++++++++----------
>   1 file changed, 27 insertions(+), 10 deletions(-)
> 
> diff --git a/kernel/hung_task.c b/kernel/hung_task.c
> index d2254c91450b..d5109a0994c5 100644
> --- a/kernel/hung_task.c
> +++ b/kernel/hung_task.c
> @@ -223,6 +223,32 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti
>   }
>   #endif
>   
> +/**
> + * hung_task_diagnostics - Print structured diagnostic info for a hung task.
> + * @t: The struct task_struct of the detected hung task.
> + *
> + * This function consolidates the printing of core diagnostic information
> + * for a task found to be blocked. This approach ensures atomic logging
> + * of the multi-line message block, preventing interleaving by other
> + * console activity, thus maintaining contextual clarity.
> + */
> +static void hung_task_diagnostics(struct task_struct *t)
> +{
> +	unsigned long blocked_secs = (jiffies - t->last_switch_time) / HZ;
> +
> +	pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n"
> +	       "      %s %s %.*s\n"
> +	       "%s\n"

"%s\n" forces an unconditional newline, causing a spurious blank line when
the flag isn't set, right?

> +	       "\"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\""
> +	       " disables this message.\n",
> +	       t->comm, t->pid, blocked_secs,
> +	       print_tainted(), init_utsname()->release,
> +	       (int)strcspn(init_utsname()->version, " "),
> +	       init_utsname()->version,
> +	       (t->flags & PF_POSTCOREDUMP) ?
> +	       "      Blocked by coredump." : "");
> +}
> +
>   static void check_hung_task(struct task_struct *t, unsigned long timeout,
>   		unsigned long prev_detect_count)
>   {
> @@ -252,16 +278,7 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout,
>   	if (sysctl_hung_task_warnings || hung_task_call_panic) {
>   		if (sysctl_hung_task_warnings > 0)
>   			sysctl_hung_task_warnings--;
> -		pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n",
> -		       t->comm, t->pid, (jiffies - t->last_switch_time) / HZ);
> -		pr_err("      %s %s %.*s\n",
> -			print_tainted(), init_utsname()->release,
> -			(int)strcspn(init_utsname()->version, " "),
> -			init_utsname()->version);
> -		if (t->flags & PF_POSTCOREDUMP)
> -			pr_err("      Blocked by coredump.\n");
> -		pr_err("\"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\""
> -			" disables this message.\n");
> +		hung_task_diagnostics(t);

Since there's only one caller, why not just open-code it ...

TBH, jamming everything into a single pr_err() call doesn't look any
cleaner to me than the original code :(

>   		sched_show_task(t);
>   		debug_show_blocker(t, timeout);
>   

Cheers,
Lance

Re: [PATCH 1/2] hung_task: Consolidate hung task warning into an atomic log block

[Greg KH]

On Tue, Dec 09, 2025 at 01:12:14PM +0800, Lance Yang wrote:
> 
> 
> On 2025/12/9 12:12, Aaron Tomlin wrote:
> > Consolidate the multi-line console output in check_hung_task() into a new
> > helper function, hung_task_diagnostics().
> > 
> > This patch ensures the entire diagnostic block (task info, kernel
> > version, and sysctl advice) is logged to the ring buffer via a single
> > pr_err() call. This is critical in a concurrent environment to prevent
> > message lines from interleaving with other CPU activity, thus
> > maintaining contextual integrity of the warning message.
> > 
> > Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
> > ---
> >   kernel/hung_task.c | 37 +++++++++++++++++++++++++++----------
> >   1 file changed, 27 insertions(+), 10 deletions(-)
> > 
> > diff --git a/kernel/hung_task.c b/kernel/hung_task.c
> > index d2254c91450b..d5109a0994c5 100644
> > --- a/kernel/hung_task.c
> > +++ b/kernel/hung_task.c
> > @@ -223,6 +223,32 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti
> >   }
> >   #endif
> > +/**
> > + * hung_task_diagnostics - Print structured diagnostic info for a hung task.
> > + * @t: The struct task_struct of the detected hung task.
> > + *
> > + * This function consolidates the printing of core diagnostic information
> > + * for a task found to be blocked. This approach ensures atomic logging
> > + * of the multi-line message block, preventing interleaving by other
> > + * console activity, thus maintaining contextual clarity.
> > + */
> > +static void hung_task_diagnostics(struct task_struct *t)
> > +{
> > +	unsigned long blocked_secs = (jiffies - t->last_switch_time) / HZ;
> > +
> > +	pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n"
> > +	       "      %s %s %.*s\n"
> > +	       "%s\n"
> 
> "%s\n" forces an unconditional newline, causing a spurious blank line when
> the flag isn't set, right?

The first \n should not be there, this should be all one line.

If you have multiple lines wanting to be printed, use multiple pr_err()
calls.

thanks,

greg k-h

Re: [PATCH 1/2] hung_task: Consolidate hung task warning into an atomic log block

[Aaron Tomlin]

[-- Attachment #1: Type: text/plain, Size: 666 bytes --]

On Tue, Dec 09, 2025 at 03:56:58PM +0900, Greg KH wrote:
> > "%s\n" forces an unconditional newline, causing a spurious blank line when
> > the flag isn't set, right?
> 
> The first \n should not be there, this should be all one line.

Hi Greg,

I agree. This will be resolved in a subsequent patch.

> If you have multiple lines wanting to be printed, use multiple pr_err()
> calls.

Unfortunately, sequential pr_err() calls risk having their lines separated
by other log messages, rendering the warning incoherent. The single-call
implementation is necessary to ensure the entire message is atomically
written.

Kind regards,
-- 
Aaron Tomlin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 1/2] hung_task: Consolidate hung task warning into an atomic log block

[Greg KH]

On Tue, Dec 09, 2025 at 05:14:40PM -0500, Aaron Tomlin wrote:
> On Tue, Dec 09, 2025 at 03:56:58PM +0900, Greg KH wrote:
> > > "%s\n" forces an unconditional newline, causing a spurious blank line when
> > > the flag isn't set, right?
> > 
> > The first \n should not be there, this should be all one line.
> 
> Hi Greg,
> 
> I agree. This will be resolved in a subsequent patch.
> 
> > If you have multiple lines wanting to be printed, use multiple pr_err()
> > calls.
> 
> Unfortunately, sequential pr_err() calls risk having their lines separated
> by other log messages, rendering the warning incoherent. The single-call
> implementation is necessary to ensure the entire message is atomically
> written.

Then perhaps you all shouldn't be dumping all that amount of information
if that's the only api you all are attempting to rely on here?

What is the goal, this should just be debugging information, no tool
should be requiring this, right?

thanks,

greg k-h

Re: [PATCH 1/2] hung_task: Consolidate hung task warning into an atomic log block

[Petr Mladek]

On Tue 2025-12-09 17:14:40, Aaron Tomlin wrote:
> On Tue, Dec 09, 2025 at 03:56:58PM +0900, Greg KH wrote:
> > > "%s\n" forces an unconditional newline, causing a spurious blank line when
> > > the flag isn't set, right?
> > 
> > The first \n should not be there, this should be all one line.
> 
> Hi Greg,
> 
> I agree. This will be resolved in a subsequent patch.

I am not sure what exactly is requested here. But a multiline message
defined on a single line would be a mess. I mean that

   printk("aaaa\nbbbbb\ncccccc\n") is hard to follow, in compare with

   printk("aaaa\n"
	  "bbbbb\n"
	  "cccccc\n");

IMHO, the only motivation to keep printk() message on a single line is
that it helps grepping. But the "\n" breaks the grepping anyway.


> > If you have multiple lines wanting to be printed, use multiple pr_err()
> > calls.
> 
> Unfortunately, sequential pr_err() calls risk having their lines separated
> by other log messages, rendering the warning incoherent. The single-call
> implementation is necessary to ensure the entire message is atomically
> written.

With the "printk" maintainer hat on, I am a bit afraid to go this way.
printk() is not designed for huge text blobs. Especially:

   + A single record is flushed to a console atomically, often
     under spin lock with IRQs disabled. Too long messages might
     cause soft lockups on slow serial consoles. [*]

   + A single record can't be wrapped at the end of the data buffer.
     Instead, the remaining space gets unused and the message is
     stored from the beginning of the buffer. So, bigger messages
     might cause more unused space. Note that this limitation
     simplified the already complicated lock-less code.

Instead, the caller ID should allow to sort the messages by a post
processing, see CONFIG_PRINTK_CALLER.

[*] The single message does not make a big difference in this case
    because the hung task report is done from atomic context. And
    it is an emergency message so that even nbcon console drivers
    try to flush it immediately.

Best Regards,
Petr

Re: [PATCH 1/2] hung_task: Consolidate hung task warning into an atomic log block

[Aaron Tomlin]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]

On Tue, Dec 09, 2025 at 01:12:14PM +0800, Lance Yang wrote:
> > +	       "      %s %s %.*s\n"
> > +	       "%s\n"
> 
> "%s\n" forces an unconditional newline, causing a spurious blank line when
> the flag isn't set, right?

Yes, apologies. I will resolve the above in a subsequent patch.

> Since there's only one caller, why not just open-code it ...

Acknowledged.

> TBH, jamming everything into a single pr_err() call doesn't look any
> cleaner to me than the original code :(

Agreed on the aesthetics; however, the technical motivation is sound. This
method is required to generate a non-interleaved diagnostic block and avoid
losing contextual information in high-concurrency logging. This is an area
of concern for us.

Kind regards,
-- 
Aaron Tomlin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH 2/2] hung_task: Provide runtime reset interface for hung task detector

[Aaron Tomlin]

This patch introduces a new write-only sysfs file,
hung_task_detect_count_reset, directly under /sys/kernel/.
This file exposes an explicit control point for resetting the
/proc/sys/kernel/hung_task_detect_count counter.

Currently, this counter persists across the lifetime of the running system.
The new interface allows administrators to reset the count after investigating
an incident without requiring a full system reboot, making it easier to track
new hung task events after intervention.

The store handler enforces strict input validation, only accepting the value
"1" to execute the reset operation.

Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 .../sysfs-kernel-hung_task_detect_count_reset |  8 +++++
 kernel/hung_task.c                            | 31 +++++++++++++++++++
 2 files changed, 39 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-kernel-hung_task_detect_count_reset

diff --git a/Documentation/ABI/testing/sysfs-kernel-hung_task_detect_count_reset b/Documentation/ABI/testing/sysfs-kernel-hung_task_detect_count_reset
new file mode 100644
index 000000000000..41dd30171c9c
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-kernel-hung_task_detect_count_reset
@@ -0,0 +1,8 @@
+What:		/sys/kernel/hung_task_detect_count_reset
+Date:		Dec 2025
+KernelVersion:	6.19
+Contact:	Linux kernel mailing list <linux-kernel@vger.kernel.org>
+Description:
+        A write-only interface to reset the persistent counter of tasks
+        detected as hung since boot. Write 1 to clear the counter and
+        restart diagnostic tracking without rebooting.
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index d5109a0994c5..035652dabf10 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -539,6 +539,37 @@ static int watchdog(void *dummy)
 	return 0;
 }
 
+#ifdef CONFIG_SYSFS
+static ssize_t hung_task_detect_count_reset_store(struct kobject *kobj,
+						  struct kobj_attribute *attr,
+						  const char *buf, size_t count)
+{
+	unsigned long val;
+	int ret;
+
+	ret = kstrtoul(buf, 0, &val);
+	if (ret)
+		return ret;
+	if (val != 1)
+		return -EINVAL;
+
+	WRITE_ONCE(sysctl_hung_task_detect_count, 0);
+
+	return count;
+}
+
+static struct kobj_attribute hung_task_detect_count_reset_attr = __ATTR_WO(hung_task_detect_count_reset);
+
+static __init int hung_task_detect_sysfs_init(void)
+{
+	sysfs_add_file_to_group(kernel_kobj,
+				&hung_task_detect_count_reset_attr.attr,
+				NULL);
+	return 0;
+}
+late_initcall(hung_task_detect_sysfs_init);
+#endif
+
 static int __init hung_task_init(void)
 {
 	atomic_notifier_chain_register(&panic_notifier_list, &panic_block);
-- 
2.51.0

Re: [PATCH 2/2] hung_task: Provide runtime reset interface for hung task detector

[Lance Yang]

On 2025/12/9 12:12, Aaron Tomlin wrote:
> This patch introduces a new write-only sysfs file,
> hung_task_detect_count_reset, directly under /sys/kernel/.
> This file exposes an explicit control point for resetting the
> /proc/sys/kernel/hung_task_detect_count counter.

Is the new file really necessary? I'd perfer if we just made the
existing hung_task_detect_count writable ;)

Just let users echo 0 (or anything) to reset it, and make sure to
update the documentation as well.

Cheers,
Lance

Re: [PATCH 2/2] hung_task: Provide runtime reset interface for hung task detector

[Aaron Tomlin]

On Tue, Dec 09, 2025 at 12:54:14PM +0800, Lance Yang wrote:
> On 2025/12/9 12:12, Aaron Tomlin wrote:
> > This patch introduces a new write-only sysfs file,
> > hung_task_detect_count_reset, directly under /sys/kernel/.
> > This file exposes an explicit control point for resetting the
> > /proc/sys/kernel/hung_task_detect_count counter.

Hi Lance,

> Is the new file really necessary? I'd perfer if we just made the
> existing hung_task_detect_count writable ;)

Acknowledged. The primary motivation here was initiating a migration away
from the legacy procfs interface.

> Just let users echo 0 (or anything) to reset it, and make sure to
> update the documentation as well.

Understood. I will send a v2.

Kind regards,
-- 
Aaron Tomlin