[PATCH v8 0/2] hung_task: Improve warning budget handling and task reporting

[PATCH v8 0/2] hung_task: Improve warning budget handling and task reporting

[Aaron Tomlin]

The hung_task watchdog detects tasks stuck in TASK_UNINTERRUPTIBLE (D)
state for longer than CONFIG_DEFAULT_HUNG_TASK_TIMEOUT seconds. To prevent
log spam during system spikes, sysctl_hung_task_warnings enforces a budget
on the number of logged warnings.

However, the current implementation has two major limitations:

    1. Permanent exhaustion of warning budget

       sysctl_hung_task_warnings is decremented directly when printing
       warnings. Once this budget hits zero, no further warnings are
       reported until an administrator manually updates the sysctl value or
       reboots the system. Consequently, a single temporary hang episode
       permanently blinds the kernel watchdog to any subsequent hung tasks
       after system recovery.

    2. Total log suppression when budget is exhausted

       Once the warning budget reaches zero, hung_task_info() completely
       suppresses all output, including the basic single-line alert. While
       suppressing verbose stack dumps and lock debugging is desirable to
       prevent dmesg flooding, hiding basic task alerts leaves
       administrators entirely unaware that tasks are hanging.

This patch series resolves both limitations by decoupling the configured
warning budget from the runtime warning counter, automatically resetting
the budget when the system recovers, and keeping basic single-line hung
task alerts visible.

Patch 1 decouples the user-configured limit sysctl_hung_task_warnings from
the runtime counter hung_task_warnings_printed, automatically resetting
hung_task_warnings_printed back to the configured limit whenever a check
interval passes with zero hung tasks detected. The runtime counter is also
kept synchronised whenever the sysctl parameter is modified, and the
corresponding sysctl documentation
(i.e., Documentation/admin-guide/sysctl/kernel.rst) is updated to reflect
this auto-reset behaviour.

Patch 2 ensures that the basic single-line hung task messsage is always
logged regardless of warning budget exhaustion, while restricting warning
budget enforcement solely to verbose diagnostics such as process stack
dumps, taint and release information, and lock blocker details.
Additionally, it updates the warning exhaustion log message to clarify to
administrators that future reports will only omit detailed process dumps
rather than being completely suppressed.

Changes since v7:

 - Consolidated the commit message of each patch (Lance Yang)

 - Linked to v7: https://lore.kernel.org/lkml/20260804155406.254810-1-atomlin@atomlin.com/

Changes since v6:

 - Restructured the series in a new direction. Introduced an internal
   counter (hung_task_warnings_printed) decoupled from
   sysctl_hung_task_warnings. The warning budget automatically resets to
   the configured limit once a check interval completes with zero hung
   tasks detected, or when modified via sysctl (Petr Mladek)

 - Ensured the single-line blocked hung task message is always printed even
   after the warning budget reaches zero. Now budget enforcement is
   restricted solely to suppressing verbose diagnostics (Petr Mladek)

 - Refined the description of sysctl hung_task_warnings (Lance Yang)

 - Removed field hung_task_reported from struct task_struct

 - Removed the CONFIG_DETECT_HUNG_TASK_BLOCKER integration and
   hung_task_blockers[] used to track memory addresses of blocker locks
   across check intervals

 - Removed the skip_show_task logic and dmesg log suppression messages

 - Removed tracking variables (warnings_decremented and
   hung_task_has_active) and the dmesg recovery notice printed when
   clearing the blocker array

 - Linked to v6: https://lore.kernel.org/lkml/20260719161305.428947-1-atomlin@atomlin.com/

Changes since v5:

 - Skipped hung_task_info() and sys_info() for tasks already reported in
   previous rounds to avoid log spam

 - Linked to v5: https://lore.kernel.org/lkml/20260712202100.123934-1-atomlin@atomlin.com/

Changes since v4:

 - Replaced the stack-local hashmap implementation with a persistent
   array (Petr Mladek)

 - Persistently track blocker addresses across scan intervals. Suppress
   warning reports and keep the sysctl_hung_task_warnings budget intact
   if the blocker is already tracked in the array (Petr Mladek)

 - Reset the warnings budget and clear the blocker array when the hang
   resolves (Petr Mladek)

 - Output a recovery message to the kernel ring buffer upon hang
   resolution

 - Linked to v4: https://lore.kernel.org/lkml/20260627205733.90983-1-atomlin@atomlin.com/

Changes since v3:

 - Deduct from the global budget if printing a full stack trace

 - Pivoted from heuristic Wait Channel hashing to deterministic
   blocker address hashing via CONFIG_DETECT_HUNG_TASK_BLOCKER

 - Replaced the hung_task_reported bit-field with a standalone u8 byte.
   Move hung_task_reported into an existing structural alignment hole
   within task_struct following blocked_lock, resulting in zero overall
   memory footprint increase and optimal cacheline grouping

 - Linked to v3: https://lore.kernel.org/lkml/20260621213756.43225-1-atomlin@atomlin.com/

Changes since v2:

 - Replaced the per-round cache flush with a task_struct bit-field for
   persistent cross-scan tracking, mitigating delayed budget exhaustion

 - Abandoned exact-stack hashing in favour of Wait Channel hashing

 - Transitioned from jhash() to hash_long() to optimise single-pointer
   hashing, and relocated the hash map to the local stack

 - Linked to v2: https://lore.kernel.org/lkml/20260620013559.1537893-1-atomlin@atomlin.com/

Changes since v1:

 - Preserve "INFO:" headers for all hung tasks; suppress only the stack
   dumps for duplicates (Masami Hiramatsu)

 - Print a clear notification when a trace is explicitly suppressed

 - Add #ifdef CONFIG_STACKTRACE guards to prevent Kconfig build errors

 - Optimise overhead by unwinding the stack only if a warning is
   actually going to be printed

 - Linked to v1: https://lore.kernel.org/lkml/20260617184841.1447955-1-atomlin@atomlin.com/

Aaron Tomlin (2):
  hung_task: Reset warning budget when problem gets resolved
  hung_task: Always print basic hung task info header

 Documentation/admin-guide/sysctl/kernel.rst |  5 ++-
 kernel/hung_task.c                          | 46 +++++++++++++++------
 2 files changed, 37 insertions(+), 14 deletions(-)

-- 
2.55.0

[PATCH v8 1/2] hung_task: Reset warning budget when problem gets resolved

[Aaron Tomlin]

sysctl_hung_task_warnings currently holds both the configured warning
limit and the remaining budget. Each detailed report decrements the
sysctl, so once it reaches zero, the configured limit is lost and cannot
be restored automatically.

Keep sysctl_hung_task_warnings unchanged and track the remaining budget
in hung_task_warnings_printed. Reset the runtime budget when a watchdog
check sees no hung tasks or when userspace writes a new sysctl value.

Assisted-by: Antigravity:gemini-3.5-flash
Suggested-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Tested-by: Lance Yang <lance.yang@linux.dev>
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 Documentation/admin-guide/sysctl/kernel.rst |  5 +--
 kernel/hung_task.c                          | 34 ++++++++++++++++-----
 2 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index c6994e55d141..b0f8e55efc1a 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -459,8 +459,9 @@ hung_task_warnings
 ==================
 
 The maximum number of warnings to report. During a check interval
-if a hung task is detected, this value is decreased by 1.
-When this value reaches 0, no more warnings will be reported.
+if a hung task is detected, the internal warning budget is decreased by 1.
+When this budget reaches 0, no more detailed warnings will be reported. The
+warning budget is reset to the configured limit when no hung task is found.
 This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.
 
 -1: report an infinite number of warnings.
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 6fcc94ce4ca9..4e1fb0db79d1 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -59,6 +59,8 @@ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs;
 
 static int __read_mostly sysctl_hung_task_warnings = 10;
 
+static int hung_task_warnings_printed = 10;
+
 static int __read_mostly did_panic;
 static bool hung_task_call_panic;
 
@@ -247,9 +249,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
 	 * accordingly
 	 */
-	if (sysctl_hung_task_warnings || hung_task_call_panic) {
-		if (sysctl_hung_task_warnings > 0)
-			sysctl_hung_task_warnings--;
+	if (hung_task_warnings_printed || hung_task_call_panic) {
+		if (hung_task_warnings_printed > 0)
+			hung_task_warnings_printed--;
 		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
 		       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
 		       (jiffies - t->last_switch_time) / HZ);
@@ -264,7 +266,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		sched_show_task(t);
 		debug_show_blocker(t, timeout);
 
-		if (!sysctl_hung_task_warnings)
+		if (!hung_task_warnings_printed)
 			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
 	}
 
@@ -304,7 +306,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	unsigned long last_break = jiffies;
 	struct task_struct *g, *t;
 	unsigned long this_round_count;
-	int need_warning = sysctl_hung_task_warnings;
+	int need_warning = hung_task_warnings_printed;
 	unsigned long si_mask = hung_task_si_mask;
 
 	/*
@@ -340,8 +342,10 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
  unlock:
 	rcu_read_unlock();
 
-	if (!this_round_count)
+	if (!this_round_count) {
+		hung_task_warnings_printed = sysctl_hung_task_warnings;
 		return;
+	}
 
 	if (need_warning || hung_task_call_panic) {
 		si_mask |= SYS_INFO_LOCKS;
@@ -425,6 +429,22 @@ static int proc_dohung_task_timeout_secs(const struct ctl_table *table, int writ
 	return ret;
 }
 
+static int proc_dohung_task_warnings(const struct ctl_table *table, int write,
+				     void *buffer,
+				     size_t *lenp, loff_t *ppos)
+{
+	int ret;
+
+	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
+
+	if (ret || !write)
+		return ret;
+
+	hung_task_warnings_printed = sysctl_hung_task_warnings;
+
+	return 0;
+}
+
 /*
  * This is needed for proc_doulongvec_minmax of sysctl_hung_task_timeout_secs
  * and hung_task_check_interval_secs
@@ -480,7 +500,7 @@ static const struct ctl_table hung_task_sysctls[] = {
 		.data		= &sysctl_hung_task_warnings,
 		.maxlen		= sizeof(int),
 		.mode		= 0644,
-		.proc_handler	= proc_dointvec_minmax,
+		.proc_handler	= proc_dohung_task_warnings,
 		.extra1		= SYSCTL_NEG_ONE,
 	},
 	{
-- 
2.55.0

[PATCH v8 2/2] hung_task: Always print basic hung task info header

[Aaron Tomlin]

Once the warning budget is exhausted, hung_task_info() stops printing any
per-task message. Detection still increments hung_task_detect_count and
emits trace_sched_process_hang(), but dmesg no longer shows which task is
blocked.

Move the "INFO: task ... blocked" line outside the budget check. Keep the
rest of the per-task report budgeted and update the exhaustion message
accordingly.

Assisted-by: Antigravity:gemini-3.5-flash
Suggested-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 kernel/hung_task.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 4e1fb0db79d1..6ebb3a87ac65 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -244,17 +244,19 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		hung_task_call_panic = true;
 	}
 
+	/* Always print the blocked message */
+	pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
+	       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
+	       (jiffies - t->last_switch_time) / HZ);
+
 	/*
 	 * The given task did not get scheduled for more than
 	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
-	 * accordingly
+	 * accordingly with full details if the budget is not exhausted.
 	 */
 	if (hung_task_warnings_printed || hung_task_call_panic) {
 		if (hung_task_warnings_printed > 0)
 			hung_task_warnings_printed--;
-		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
-		       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
-		       (jiffies - t->last_switch_time) / HZ);
 		pr_err("      %s %s %.*s\n",
 			print_tainted(), init_utsname()->release,
 			(int)strcspn(init_utsname()->version, " "),
@@ -267,7 +269,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		debug_show_blocker(t, timeout);
 
 		if (!hung_task_warnings_printed)
-			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
+			pr_info("Future hung task reports won't print details about each process, see sysctl kernel.hung_task_warnings\n");
 	}
 
 	touch_nmi_watchdog();
-- 
2.55.0

Re: [PATCH v8 0/2] hung_task: Improve warning budget handling and task reporting

[Andrew Morton]

On Tue,  4 Aug 2026 16:20:48 -0400 Aaron Tomlin <atomlin@atomlin.com> wrote:

> The hung_task watchdog detects tasks stuck in TASK_UNINTERRUPTIBLE (D)
> state for longer than CONFIG_DEFAULT_HUNG_TASK_TIMEOUT seconds. To prevent
> log spam during system spikes, sysctl_hung_task_warnings enforces a budget
> on the number of logged warnings.
> 
> However, the current implementation has two major limitations:
> 
>     1. Permanent exhaustion of warning budget
> 
>        sysctl_hung_task_warnings is decremented directly when printing
>        warnings. Once this budget hits zero, no further warnings are
>        reported until an administrator manually updates the sysctl value or
>        reboots the system. Consequently, a single temporary hang episode
>        permanently blinds the kernel watchdog to any subsequent hung tasks
>        after system recovery.
> 
>     2. Total log suppression when budget is exhausted
> 
>        Once the warning budget reaches zero, hung_task_info() completely
>        suppresses all output, including the basic single-line alert. While
>        suppressing verbose stack dumps and lock debugging is desirable to
>        prevent dmesg flooding, hiding basic task alerts leaves
>        administrators entirely unaware that tasks are hanging.
> 
> This patch series resolves both limitations by decoupling the configured
> warning budget from the runtime warning counter, automatically resetting
> the budget when the system recovers, and keeping basic single-line hung
> task alerts visible.

Thanks.  A couple of concerns from AI review:
	https://sashiko.dev/#/patchset/20260804202050.262427-1-atomlin@atomlin.com

Apologies if these were considered during review of previous
iterations.

Re: [PATCH v8 0/2] hung_task: Improve warning budget handling and task reporting

[Aaron Tomlin]

On Wed, Aug 05, 2026 at 10:13:05AM +0800, Lance Yang wrote:
> 
> 
> On 2026/8/5 07:05, Andrew Morton wrote:
> > On Tue,  4 Aug 2026 16:20:48 -0400 Aaron Tomlin <atomlin@atomlin.com> wrote:
> > 
> > > The hung_task watchdog detects tasks stuck in TASK_UNINTERRUPTIBLE (D)
> > > state for longer than CONFIG_DEFAULT_HUNG_TASK_TIMEOUT seconds. To prevent
> > > log spam during system spikes, sysctl_hung_task_warnings enforces a budget
> > > on the number of logged warnings.
> > > 
> > > However, the current implementation has two major limitations:
> > > 
> > >      1. Permanent exhaustion of warning budget
> > > 
> > >         sysctl_hung_task_warnings is decremented directly when printing
> > >         warnings. Once this budget hits zero, no further warnings are
> > >         reported until an administrator manually updates the sysctl value or
> > >         reboots the system. Consequently, a single temporary hang episode
> > >         permanently blinds the kernel watchdog to any subsequent hung tasks
> > >         after system recovery.
> > > 
> > >      2. Total log suppression when budget is exhausted
> > > 
> > >         Once the warning budget reaches zero, hung_task_info() completely
> > >         suppresses all output, including the basic single-line alert. While
> > >         suppressing verbose stack dumps and lock debugging is desirable to
> > >         prevent dmesg flooding, hiding basic task alerts leaves
> > >         administrators entirely unaware that tasks are hanging.
> > > 
> > > This patch series resolves both limitations by decoupling the configured
> > > warning budget from the runtime warning counter, automatically resetting
> > > the budget when the system recovers, and keeping basic single-line hung
> > > task alerts visible.
> > 
> > Thanks.  A couple of concerns from AI review:
> > 	https://sashiko.dev/#/patchset/20260804202050.262427-1-atomlin@atomlin.com
> 
> I'm not quite sure what the cleanest way to handle these is yet, but
> both points look fair.
> 
> 1) Concurrent writes to hung_task_warnings can race and leave
>    hung_task_warnings_printed out of sync with it.
> 
> 2) The unconditional pr_err() is also no longer bounded by
>    hung_task_warnings. With lots of hung tasks, every scan can flood
>    the log and console with one line per task. Maybe rate-limit those
>    messages or cap them per scan.
> 
> > Apologies if these were considered during review of previous
> > iterations.

Hi Andrew, Lance,

Yes. However, I feel the first one is of a lesser concern. For instance,
consider the following race scenario, when two threads write to the sysctl
concurrently:

    - Thread A writes value 10, 'writes sysctl_hung_task_warnings = 10'
    - Thread B writes value 20, 'writes sysctl_hung_task_warnings = 20'
    - Thread B executes 'hung_task_warnings_printed =
      sysctl_hung_task_warnings' (i.e., sets 20)

    - Thread A resumes and executes 'hung_task_warnings_printed =
      sysctl_hung_task_warnings' using its _cached_ register value 10

The result, sysctl_hung_task_warnings holds 20, but
hung_task_warnings_printed holds 10.

I suspect the severity is low since concurrent sysctl writes are likely
rare—restricted to CAP_SYS_ADMIN. Finally, if de-synchronisation occurs,
the system automatically self-heals as soon as a watchdog check finds zero
hung tasks (this_round_count == 0), resetting hung_task_warnings_printed =
sysctl_hung_task_warnings.

However, I would rather not leave the data race unresolved. How about using
READ_ONCE() and WRITE_ONCE()? I think multi-variable transactional
atomicity is unnecessary:

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 4e1fb0db79d1..123456789abc 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -348,7 +349,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)

     if (!this_round_count) {
-        hung_task_warnings_printed = sysctl_hung_task_warnings;
+        WRITE_ONCE(hung_task_warnings_printed, READ_ONCE(sysctl_hung_task_warnings));
         return;
     }

@@ -429,14 +430,14 @@ static int proc_dohung_task_warnings(const struct ctl_table *table, int write,
                      void *buffer,
                      size_t *lenp, loff_t *ppos)
 {
     int ret;

     ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);

     if (ret || !write)
         return ret;

-    hung_task_warnings_printed = sysctl_hung_task_warnings;
+    WRITE_ONCE(hung_task_warnings_printed, READ_ONCE(sysctl_hung_task_warnings));

     return 0;
 }

For the second issue, this is very serious. We could move the per-task
blocked message back inside the budget check and when the budget is
exhausted, emit a single aggregate summary line at the end of
check_hung_uninterruptible_tasks(). However, this is not ideal.

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 6ebb3a87ac65..b0932a921d7b 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -244,17 +244,17 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
         hung_task_call_panic = true;
     }

-    /* Always print the blocked message */
-    pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
-           t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
-           (jiffies - t->last_switch_time) / HZ);
-
     /*
      * The given task did not get scheduled for more than
      * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
      * accordingly with full details if the budget is not exhausted.
      */
     if (hung_task_warnings_printed || hung_task_call_panic) {
         if (hung_task_warnings_printed > 0)
             hung_task_warnings_printed--;
+        pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
+               t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
+               (jiffies - t->last_switch_time) / HZ);
         pr_err("      %s %s %.*s\n",
             print_tainted(), init_utsname()->release,
             (int)strcspn(init_utsname()->version, " "),
@@ -348,6 +348,11 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)

     if (!this_round_count) {
         hung_task_warnings_printed = sysctl_hung_task_warnings;
         return;
     }

+    if (!hung_task_warnings_printed && !hung_task_call_panic) {
+        pr_info("khungtaskd: %lu tasks blocked for more than %ld seconds (warning budget exhausted)\n",
+            this_round_count, timeout);
+    }


Kind regards,
-- 
Aaron Tomlin

Re: [PATCH v8 0/2] hung_task: Improve warning budget handling and task reporting

[Lance Yang]

On 2026/8/5 22:16, Aaron Tomlin wrote:
> On Wed, Aug 05, 2026 at 10:13:05AM +0800, Lance Yang wrote:
>>
>>
>> On 2026/8/5 07:05, Andrew Morton wrote:
>>> On Tue,  4 Aug 2026 16:20:48 -0400 Aaron Tomlin <atomlin@atomlin.com> wrote:
>>>
>>>> The hung_task watchdog detects tasks stuck in TASK_UNINTERRUPTIBLE (D)
>>>> state for longer than CONFIG_DEFAULT_HUNG_TASK_TIMEOUT seconds. To prevent
>>>> log spam during system spikes, sysctl_hung_task_warnings enforces a budget
>>>> on the number of logged warnings.
>>>>
>>>> However, the current implementation has two major limitations:
>>>>
>>>>       1. Permanent exhaustion of warning budget
>>>>
>>>>          sysctl_hung_task_warnings is decremented directly when printing
>>>>          warnings. Once this budget hits zero, no further warnings are
>>>>          reported until an administrator manually updates the sysctl value or
>>>>          reboots the system. Consequently, a single temporary hang episode
>>>>          permanently blinds the kernel watchdog to any subsequent hung tasks
>>>>          after system recovery.
>>>>
>>>>       2. Total log suppression when budget is exhausted
>>>>
>>>>          Once the warning budget reaches zero, hung_task_info() completely
>>>>          suppresses all output, including the basic single-line alert. While
>>>>          suppressing verbose stack dumps and lock debugging is desirable to
>>>>          prevent dmesg flooding, hiding basic task alerts leaves
>>>>          administrators entirely unaware that tasks are hanging.
>>>>
>>>> This patch series resolves both limitations by decoupling the configured
>>>> warning budget from the runtime warning counter, automatically resetting
>>>> the budget when the system recovers, and keeping basic single-line hung
>>>> task alerts visible.
>>>
>>> Thanks.  A couple of concerns from AI review:
>>> 	https://sashiko.dev/#/patchset/20260804202050.262427-1-atomlin@atomlin.com
>>
>> I'm not quite sure what the cleanest way to handle these is yet, but
>> both points look fair.
>>
>> 1) Concurrent writes to hung_task_warnings can race and leave
>>     hung_task_warnings_printed out of sync with it.
>>
>> 2) The unconditional pr_err() is also no longer bounded by
>>     hung_task_warnings. With lots of hung tasks, every scan can flood
>>     the log and console with one line per task. Maybe rate-limit those
>>     messages or cap them per scan.
>>
>>> Apologies if these were considered during review of previous
>>> iterations.
> 
> Hi Andrew, Lance,
> 
> Yes. However, I feel the first one is of a lesser concern. For instance,
> consider the following race scenario, when two threads write to the sysctl
> concurrently:
> 
>      - Thread A writes value 10, 'writes sysctl_hung_task_warnings = 10'
>      - Thread B writes value 20, 'writes sysctl_hung_task_warnings = 20'
>      - Thread B executes 'hung_task_warnings_printed =
>        sysctl_hung_task_warnings' (i.e., sets 20)
> 
>      - Thread A resumes and executes 'hung_task_warnings_printed =
>        sysctl_hung_task_warnings' using its _cached_ register value 10
> 
> The result, sysctl_hung_task_warnings holds 20, but
> hung_task_warnings_printed holds 10.
> 
> I suspect the severity is low since concurrent sysctl writes are likely
> rare—restricted to CAP_SYS_ADMIN. Finally, if de-synchronisation occurs,
> the system automatically self-heals as soon as a watchdog check finds zero
> hung tasks (this_round_count == 0), resetting hung_task_warnings_printed =
> sysctl_hung_task_warnings.
> 
> However, I would rather not leave the data race unresolved. How about using
> READ_ONCE() and WRITE_ONCE()? I think multi-variable transactional
> atomicity is unnecessary:

Doesn't close the race. A can read 10, B can finish both updates
with 20, then A writes 10 back. Still ends up 20/10.

READ_ONCE()/WRITE_ONCE() don't serialize anything here ...

> 
> diff --git a/kernel/hung_task.c b/kernel/hung_task.c
> index 4e1fb0db79d1..123456789abc 100644
> --- a/kernel/hung_task.c
> +++ b/kernel/hung_task.c
> @@ -348,7 +349,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
> 
>       if (!this_round_count) {
> -        hung_task_warnings_printed = sysctl_hung_task_warnings;
> +        WRITE_ONCE(hung_task_warnings_printed, READ_ONCE(sysctl_hung_task_warnings));
>           return;
>       }
> 
> @@ -429,14 +430,14 @@ static int proc_dohung_task_warnings(const struct ctl_table *table, int write,
>                        void *buffer,
>                        size_t *lenp, loff_t *ppos)
>   {
>       int ret;
> 
>       ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
> 
>       if (ret || !write)
>           return ret;
> 
> -    hung_task_warnings_printed = sysctl_hung_task_warnings;
> +    WRITE_ONCE(hung_task_warnings_printed, READ_ONCE(sysctl_hung_task_warnings));
> 
>       return 0;
>   }
> 
> For the second issue, this is very serious. We could move the per-task
> blocked message back inside the budget check and when the budget is
> exhausted, emit a single aggregate summary line at the end of
> check_hung_uninterruptible_tasks(). However, this is not ideal.

Yeah, aggregate summary works. I'd drop timeout, though. It's just
scan threshold, not actual blocked time, and summary no longer refers
to any one task. Maybe just:

pr_info("khungtaskd: %lu hung tasks detected (warning budget exhausted)\n",
	this_round_count);

> diff --git a/kernel/hung_task.c b/kernel/hung_task.c
> index 6ebb3a87ac65..b0932a921d7b 100644
> --- a/kernel/hung_task.c
> +++ b/kernel/hung_task.c
> @@ -244,17 +244,17 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
>           hung_task_call_panic = true;
>       }
> 
> -    /* Always print the blocked message */
> -    pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
> -           t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
> -           (jiffies - t->last_switch_time) / HZ);
> -
>       /*
>        * The given task did not get scheduled for more than
>        * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
>        * accordingly with full details if the budget is not exhausted.
>        */
>       if (hung_task_warnings_printed || hung_task_call_panic) {
>           if (hung_task_warnings_printed > 0)
>               hung_task_warnings_printed--;
> +        pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
> +               t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
> +               (jiffies - t->last_switch_time) / HZ);
>           pr_err("      %s %s %.*s\n",
>               print_tainted(), init_utsname()->release,
>               (int)strcspn(init_utsname()->version, " "),
> @@ -348,6 +348,11 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
> 
>       if (!this_round_count) {
>           hung_task_warnings_printed = sysctl_hung_task_warnings;
>           return;
>       }
> 
> +    if (!hung_task_warnings_printed && !hung_task_call_panic) {
> +        pr_info("khungtaskd: %lu tasks blocked for more than %ld seconds (warning budget exhausted)\n",
> +            this_round_count, timeout);
> +    }
> 
> 
> Kind regards,

Cheers, Lance

Re: [PATCH v8 0/2] hung_task: Improve warning budget handling and task reporting

[Aaron Tomlin]

On Thu, Aug 06, 2026 at 10:05:10AM +0800, Lance Yang wrote:
> 
> 
> On 2026/8/5 22:16, Aaron Tomlin wrote:
> > On Wed, Aug 05, 2026 at 10:13:05AM +0800, Lance Yang wrote:
> > > 
> > > 
> > > On 2026/8/5 07:05, Andrew Morton wrote:
> > > > On Tue,  4 Aug 2026 16:20:48 -0400 Aaron Tomlin <atomlin@atomlin.com> wrote:
> > > > 
> > > > > The hung_task watchdog detects tasks stuck in TASK_UNINTERRUPTIBLE (D)
> > > > > state for longer than CONFIG_DEFAULT_HUNG_TASK_TIMEOUT seconds. To prevent
> > > > > log spam during system spikes, sysctl_hung_task_warnings enforces a budget
> > > > > on the number of logged warnings.
> > > > > 
> > > > > However, the current implementation has two major limitations:
> > > > > 
> > > > >       1. Permanent exhaustion of warning budget
> > > > > 
> > > > >          sysctl_hung_task_warnings is decremented directly when printing
> > > > >          warnings. Once this budget hits zero, no further warnings are
> > > > >          reported until an administrator manually updates the sysctl value or
> > > > >          reboots the system. Consequently, a single temporary hang episode
> > > > >          permanently blinds the kernel watchdog to any subsequent hung tasks
> > > > >          after system recovery.
> > > > > 
> > > > >       2. Total log suppression when budget is exhausted
> > > > > 
> > > > >          Once the warning budget reaches zero, hung_task_info() completely
> > > > >          suppresses all output, including the basic single-line alert. While
> > > > >          suppressing verbose stack dumps and lock debugging is desirable to
> > > > >          prevent dmesg flooding, hiding basic task alerts leaves
> > > > >          administrators entirely unaware that tasks are hanging.
> > > > > 
> > > > > This patch series resolves both limitations by decoupling the configured
> > > > > warning budget from the runtime warning counter, automatically resetting
> > > > > the budget when the system recovers, and keeping basic single-line hung
> > > > > task alerts visible.
> > > > 
> > > > Thanks.  A couple of concerns from AI review:
> > > > 	https://sashiko.dev/#/patchset/20260804202050.262427-1-atomlin@atomlin.com
> > > 
> > > I'm not quite sure what the cleanest way to handle these is yet, but
> > > both points look fair.
> > > 
> > > 1) Concurrent writes to hung_task_warnings can race and leave
> > >     hung_task_warnings_printed out of sync with it.
> > > 
> > > 2) The unconditional pr_err() is also no longer bounded by
> > >     hung_task_warnings. With lots of hung tasks, every scan can flood
> > >     the log and console with one line per task. Maybe rate-limit those
> > >     messages or cap them per scan.
> > > 
> > > > Apologies if these were considered during review of previous
> > > > iterations.
> > 
> > Hi Andrew, Lance,
> > 
> > Yes. However, I feel the first one is of a lesser concern. For instance,
> > consider the following race scenario, when two threads write to the sysctl
> > concurrently:
> > 
> >      - Thread A writes value 10, 'writes sysctl_hung_task_warnings = 10'
> >      - Thread B writes value 20, 'writes sysctl_hung_task_warnings = 20'
> >      - Thread B executes 'hung_task_warnings_printed =
> >        sysctl_hung_task_warnings' (i.e., sets 20)
> > 
> >      - Thread A resumes and executes 'hung_task_warnings_printed =
> >        sysctl_hung_task_warnings' using its _cached_ register value 10
> > 
> > The result, sysctl_hung_task_warnings holds 20, but
> > hung_task_warnings_printed holds 10.
> > 
> > I suspect the severity is low since concurrent sysctl writes are likely
> > rare—restricted to CAP_SYS_ADMIN. Finally, if de-synchronisation occurs,
> > the system automatically self-heals as soon as a watchdog check finds zero
> > hung tasks (this_round_count == 0), resetting hung_task_warnings_printed =
> > sysctl_hung_task_warnings.
> > 
> > However, I would rather not leave the data race unresolved. How about using
> > READ_ONCE() and WRITE_ONCE()? I think multi-variable transactional
> > atomicity is unnecessary:
> 
> Doesn't close the race. A can read 10, B can finish both updates
> with 20, then A writes 10 back. Still ends up 20/10.
> 
> READ_ONCE()/WRITE_ONCE() don't serialize anything here ...

Yes; READ_ONCE() and WRITE_ONCE() guarantee single-copy load/store
atomicity to prevent compiler optimisations. However, I agree, they do not
establish a critical section or serialise multi-step operations across
variables 'sysctl_hung_task_warnings' and 'hung_task_warnings_printed'.
How about the following?

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 6ebb3a87ac65..a977dc280262 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -237,6 +237,8 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti
 static void hung_task_info(struct task_struct *t, unsigned long timeout,
 			   unsigned long this_round_count)
 {
+	int warnings;
+
 	trace_sched_process_hang(t);
 
 	if (sysctl_hung_task_panic && this_round_count >= sysctl_hung_task_panic) {
@@ -254,9 +256,11 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
 	 * accordingly with full details if the budget is not exhausted.
 	 */
-	if (hung_task_warnings_printed || hung_task_call_panic) {
-		if (hung_task_warnings_printed > 0)
-			hung_task_warnings_printed--;
+	warnings = READ_ONCE(hung_task_warnings_printed);
+
+	if (warnings || hung_task_call_panic) {
+		if (warnings > 0)
+			WRITE_ONCE(hung_task_warnings_printed, warnings - 1);
 		pr_err("      %s %s %.*s\n",
 			print_tainted(), init_utsname()->release,
 			(int)strcspn(init_utsname()->version, " "),
@@ -268,7 +272,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		sched_show_task(t);
 		debug_show_blocker(t, timeout);
 
-		if (!hung_task_warnings_printed)
+		if (!READ_ONCE(hung_task_warnings_printed))
 			pr_info("Future hung task reports won't print details about each process, see sysctl kernel.hung_task_warnings\n");
 	}
 
@@ -308,7 +312,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	unsigned long last_break = jiffies;
 	struct task_struct *g, *t;
 	unsigned long this_round_count;
-	int need_warning = hung_task_warnings_printed;
+	int need_warning = READ_ONCE(hung_task_warnings_printed);
 	unsigned long si_mask = hung_task_si_mask;
 
 	/*
@@ -345,7 +349,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	rcu_read_unlock();
 
 	if (!this_round_count) {
-		hung_task_warnings_printed = sysctl_hung_task_warnings;
+		WRITE_ONCE(hung_task_warnings_printed, READ_ONCE(sysctl_hung_task_warnings));
 		return;
 	}
 
@@ -431,20 +435,21 @@ static int proc_dohung_task_timeout_secs(const struct ctl_table *table, int writ
 	return ret;
 }
 
+static DEFINE_MUTEX(hung_task_sysctl_mutex);
+
 static int proc_dohung_task_warnings(const struct ctl_table *table, int write,
 				     void *buffer,
 				     size_t *lenp, loff_t *ppos)
 {
 	int ret;
 
+	mutex_lock(&hung_task_sysctl_mutex);
 	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
+	if (!ret && write)
+		WRITE_ONCE(hung_task_warnings_printed, READ_ONCE(sysctl_hung_task_warnings));
+	mutex_unlock(&hung_task_sysctl_mutex);
 
-	if (ret || !write)
-		return ret;
-
-	hung_task_warnings_printed = sysctl_hung_task_warnings;
-
-	return 0;
+	return ret;
 }
 
 /*


> > diff --git a/kernel/hung_task.c b/kernel/hung_task.c
> > index 4e1fb0db79d1..123456789abc 100644
> > --- a/kernel/hung_task.c
> > +++ b/kernel/hung_task.c
> > @@ -348,7 +349,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
> > 
> >       if (!this_round_count) {
> > -        hung_task_warnings_printed = sysctl_hung_task_warnings;
> > +        WRITE_ONCE(hung_task_warnings_printed, READ_ONCE(sysctl_hung_task_warnings));
> >           return;
> >       }
> > 
> > @@ -429,14 +430,14 @@ static int proc_dohung_task_warnings(const struct ctl_table *table, int write,
> >                        void *buffer,
> >                        size_t *lenp, loff_t *ppos)
> >   {
> >       int ret;
> > 
> >       ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
> > 
> >       if (ret || !write)
> >           return ret;
> > 
> > -    hung_task_warnings_printed = sysctl_hung_task_warnings;
> > +    WRITE_ONCE(hung_task_warnings_printed, READ_ONCE(sysctl_hung_task_warnings));
> > 
> >       return 0;
> >   }
> > 
> > For the second issue, this is very serious. We could move the per-task
> > blocked message back inside the budget check and when the budget is
> > exhausted, emit a single aggregate summary line at the end of
> > check_hung_uninterruptible_tasks(). However, this is not ideal.
> 
> Yeah, aggregate summary works. I'd drop timeout, though. It's just
> scan threshold, not actual blocked time, and summary no longer refers
> to any one task. Maybe just:
> 
> pr_info("khungtaskd: %lu hung tasks detected (warning budget exhausted)\n",
> 	this_round_count);

Acknowledged.


Kind regards,
-- 
Aaron Tomlin

Re: [PATCH v8 0/2] hung_task: Improve warning budget handling and task reporting

[Lance Yang]

On Thu, Aug 06, 2026 at 10:03:57AM -0400, Aaron Tomlin wrote:
>On Thu, Aug 06, 2026 at 10:05:10AM +0800, Lance Yang wrote:
>> 
>> 
>> On 2026/8/5 22:16, Aaron Tomlin wrote:
>> > On Wed, Aug 05, 2026 at 10:13:05AM +0800, Lance Yang wrote:
>> > > 
>> > > 
>> > > On 2026/8/5 07:05, Andrew Morton wrote:
>> > > > On Tue,  4 Aug 2026 16:20:48 -0400 Aaron Tomlin <atomlin@atomlin.com> wrote:
>> > > > 
[...]
>> > > > 
>> > > > Thanks.  A couple of concerns from AI review:
>> > > > 	https://sashiko.dev/#/patchset/20260804202050.262427-1-atomlin@atomlin.com
>> > > 
>> > > I'm not quite sure what the cleanest way to handle these is yet, but
>> > > both points look fair.
>> > > 
>> > > 1) Concurrent writes to hung_task_warnings can race and leave
>> > >     hung_task_warnings_printed out of sync with it.
>> > > 
>> > > 2) The unconditional pr_err() is also no longer bounded by
>> > >     hung_task_warnings. With lots of hung tasks, every scan can flood
>> > >     the log and console with one line per task. Maybe rate-limit those
>> > >     messages or cap them per scan.
>> > > 
>> > > > Apologies if these were considered during review of previous
>> > > > iterations.
>> > 
>> > Hi Andrew, Lance,
>> > 
>> > Yes. However, I feel the first one is of a lesser concern. For instance,
>> > consider the following race scenario, when two threads write to the sysctl
>> > concurrently:
>> > 
>> >      - Thread A writes value 10, 'writes sysctl_hung_task_warnings = 10'
>> >      - Thread B writes value 20, 'writes sysctl_hung_task_warnings = 20'
>> >      - Thread B executes 'hung_task_warnings_printed =
>> >        sysctl_hung_task_warnings' (i.e., sets 20)
>> > 
>> >      - Thread A resumes and executes 'hung_task_warnings_printed =
>> >        sysctl_hung_task_warnings' using its _cached_ register value 10
>> > 
>> > The result, sysctl_hung_task_warnings holds 20, but
>> > hung_task_warnings_printed holds 10.
>> > 
>> > I suspect the severity is low since concurrent sysctl writes are likely
>> > rare—restricted to CAP_SYS_ADMIN. Finally, if de-synchronisation occurs,
>> > the system automatically self-heals as soon as a watchdog check finds zero
>> > hung tasks (this_round_count == 0), resetting hung_task_warnings_printed =
>> > sysctl_hung_task_warnings.
>> > 
>> > However, I would rather not leave the data race unresolved. How about using
>> > READ_ONCE() and WRITE_ONCE()? I think multi-variable transactional
>> > atomicity is unnecessary:
>> 
>> Doesn't close the race. A can read 10, B can finish both updates
>> with 20, then A writes 10 back. Still ends up 20/10.
>> 
>> READ_ONCE()/WRITE_ONCE() don't serialize anything here ...
>
>Yes; READ_ONCE() and WRITE_ONCE() guarantee single-copy load/store
>atomicity to prevent compiler optimisations. However, I agree, they do not
>establish a critical section or serialise multi-step operations across
>variables 'sysctl_hung_task_warnings' and 'hung_task_warnings_printed'.
>How about the following?

Still races with khungtaskd ... It can read 10, sysctl resets budget
to 0, then khungtaskd writes 9 back. We end up with sysctl at 0 and
runtime budget at 9 :)

READ_ONCE()/WRITE_ONCE() don't fix that lost update ...

Let's keep it simpler: make khungtaskd sole owner of
hung_task_warnings_printed. Sysctl write just sets a reset flag:

static atomic_t reset_hung_task_warnings = ATOMIC_INIT(0);

...

static int proc_dohung_task_warnings(const struct ctl_table *table, int write,
...
{	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
	if (!ret && write)
		atomic_set_release(&reset_hung_task_warnings, 1);
...
}

khungtaskd picks it up next scan:

	if (atomic_xchg(&reset_hung_task_warnings, 0))
		hung_task_warnings_printed =
			READ_ONCE(sysctl_hung_task_warnings);


That's it. Only khungtaskd touches runtime budget, so no lock needed
there. Write during a scan takes effect next scan, keeping whole scan
on one budget. Multiple writes collapse into one reset, and writing
same value still refills budget.

Played around with it a bit and ended up with the following on top:

---8<---
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 6ebb3a87ac65..ea7817e1856b 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -60,6 +60,7 @@ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs;
 static int __read_mostly sysctl_hung_task_warnings = 10;

 static int hung_task_warnings_printed = 10;
+static atomic_t reset_hung_task_warnings = ATOMIC_INIT(0);

 static int __read_mostly did_panic;
 static bool hung_task_call_panic;
@@ -244,11 +245,6 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		hung_task_call_panic = true;
 	}

-	/* Always print the blocked message */
-	pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
-	       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
-	       (jiffies - t->last_switch_time) / HZ);
-
 	/*
 	 * The given task did not get scheduled for more than
 	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
@@ -257,6 +253,10 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 	if (hung_task_warnings_printed || hung_task_call_panic) {
 		if (hung_task_warnings_printed > 0)
 			hung_task_warnings_printed--;
+		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
+		       t->comm, t->pid,
+		       t->in_iowait ? " in I/O wait" : "",
+		       (jiffies - t->last_switch_time) / HZ);
 		pr_err("      %s %s %.*s\n",
 			print_tainted(), init_utsname()->release,
 			(int)strcspn(init_utsname()->version, " "),
@@ -308,7 +308,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	unsigned long last_break = jiffies;
 	struct task_struct *g, *t;
 	unsigned long this_round_count;
-	int need_warning = hung_task_warnings_printed;
+	int need_warning;
 	unsigned long si_mask = hung_task_si_mask;

 	/*
@@ -318,6 +318,11 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	if (test_taint(TAINT_DIE) || did_panic)
 		return;

+	if (atomic_xchg(&reset_hung_task_warnings, 0))
+		hung_task_warnings_printed =
+			READ_ONCE(sysctl_hung_task_warnings);
+	need_warning = hung_task_warnings_printed;
+
 	this_round_count = 0;
 	rcu_read_lock();
 	for_each_process_thread(g, t) {
@@ -345,10 +350,15 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	rcu_read_unlock();

 	if (!this_round_count) {
-		hung_task_warnings_printed = sysctl_hung_task_warnings;
+		hung_task_warnings_printed =
+			READ_ONCE(sysctl_hung_task_warnings);
 		return;
 	}

+	if (!hung_task_warnings_printed && !hung_task_call_panic)
+		pr_info("khungtaskd: %lu hung tasks detected (warning budget exhausted)\n",
+			this_round_count);
+
 	if (need_warning || hung_task_call_panic) {
 		si_mask |= SYS_INFO_LOCKS;

@@ -438,13 +448,10 @@ static int proc_dohung_task_warnings(const struct ctl_table *table, int write,
 	int ret;

 	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
+	if (!ret && write)
+		atomic_set_release(&reset_hung_task_warnings, 1);

-	if (ret || !write)
-		return ret;
-
-	hung_task_warnings_printed = sysctl_hung_task_warnings;
-
-	return 0;
+	return ret;
 }

 /*
---

Cheers, Lance

[...]