[PATCH v7 0/2] hung_task: Improve warning budget handling and task reporting

[PATCH v7 0/2] hung_task: Improve warning budget handling and task reporting

[Aaron Tomlin]

The hung_task watchdog detects tasks stuck in uninterruptible (D) state for
longer than CONFIG_DEFAULT_HUNG_TASK_TIMEOUT seconds. To prevent log spam
during system spikes, sysctl_hung_task_warnings enforces a budget on the
number of logged warnings.

However, the current implementation has two major limitations:

    1. Permanent exhaustion of warning budget

       sysctl_hung_task_warnings is decremented directly when printing
       warnings. Once this budget hits zero, no further warnings are
       reported until an administrator manually updates the sysctl value or
       reboots the system. Consequently, a single temporary hang episode
       permanently blinds the kernel watchdog to any subsequent hung tasks
       after system recovery.

    2. Total log suppression when budget is exhausted

       Once the warning budget reaches zero, hung_task_info() completely
       suppresses all output, including the basic single-line alert. While
       suppressing verbose stack dumps and lock debugging is desirable to
       prevent dmesg flooding, hiding basic task alerts leaves
       administrators entirely unaware that tasks are hanging.

This patch series resolves both limitations by decoupling the configured
warning budget from the runtime warning counter, automatically resetting
the budget when the system recovers, and keeping basic single-line hung
task alerts visible.

Patch 1 decouples the user-configured limit sysctl_hung_task_warnings from
the runtime counter hung_task_warnings_printed, automatically resetting
hung_task_warnings_printed back to the configured limit whenever a check
interval passes with zero hung tasks detected. The runtime counter is also
kept synchronized whenever the sysctl parameter is modified, and the
corresponding sysctl documentation
(Documentation/admin-guide/sysctl/kernel.rst) is updated to reflect this
auto-reset behaviour.

Patch 2 ensures that the basic single-line pr_err("INFO: task %s:%d
blocked...") header is always logged regardless of warning budget
exhaustion, while restricting warning budget enforcement solely to verbose
diagnostics such as process stack dumps, taint and release information, and
lock blocker details. Additionally, it updates the warning exhaustion log
message to clarify to administrators that future reports will only omit
detailed process dumps rather than being completely suppressed.

Changes since v6:

 - Refined the description of sysctl hung_task_warnings (Lance Yang)

 - Linked to v6: https://lore.kernel.org/lkml/20260719161305.428947-1-atomlin@atomlin.com/

Changes since v5:

 - Skipped hung_task_info() and sys_info() for tasks already reported in
   previous rounds to avoid log spam

 - Linked to v5: https://lore.kernel.org/lkml/20260712202100.123934-1-atomlin@atomlin.com/

Changes since v4:

 - Replaced the stack-local hashmap implementation with a persistent
   array (Petr Mladek)

 - Persistently track blocker addresses across scan intervals. Suppress
   warning reports and keep the sysctl_hung_task_warnings budget intact
   if the blocker is already tracked in the array (Petr Mladek)

 - Reset the warnings budget and clear the blocker array when the hang
   resolves (Petr Mladek)

 - Output a recovery message to the kernel ring buffer upon hang
   resolution

 - Linked to v4: https://lore.kernel.org/lkml/20260627205733.90983-1-atomlin@atomlin.com/

Changes since v3:

 - Deduct from the global budget if printing a full stack trace

 - Pivoted from heuristic Wait Channel hashing to deterministic
   blocker address hashing via CONFIG_DETECT_HUNG_TASK_BLOCKER

 - Replaced the hung_task_reported bit-field with a standalone u8 byte.
   Move hung_task_reported into an existing structural alignment hole
   within task_struct following blocked_lock, resulting in zero overall
   memory footprint increase and optimal cacheline grouping

 - Linked to v3: https://lore.kernel.org/lkml/20260621213756.43225-1-atomlin@atomlin.com/

Changes since v2:

 - Replaced the per-round cache flush with a task_struct bit-field for
   persistent cross-scan tracking, mitigating delayed budget exhaustion

 - Abandoned exact-stack hashing in favour of Wait Channel hashing

 - Transitioned from jhash() to hash_long() to optimise single-pointer
   hashing, and relocated the hash map to the local stack

 - Linked to v2: https://lore.kernel.org/lkml/20260620013559.1537893-1-atomlin@atomlin.com/

Changes since v1:

 - Preserve "INFO:" headers for all hung tasks; suppress only the stack
   dumps for duplicates (Masami Hiramatsu)

 - Print a clear notification when a trace is explicitly suppressed

 - Add #ifdef CONFIG_STACKTRACE guards to prevent Kconfig build errors

 - Optimise overhead by unwinding the stack only if a warning is
   actually going to be printed

 - Linked to v1: https://lore.kernel.org/lkml/20260617184841.1447955-1-atomlin@atomlin.com/

Petr Mladek (2):
  hung_task: Reset warning budget when problem gets resolved
  hung_task: Always print basic hung task info header

 Documentation/admin-guide/sysctl/kernel.rst |  5 ++-
 kernel/hung_task.c                          | 46 +++++++++++++++------
 2 files changed, 37 insertions(+), 14 deletions(-)

-- 
2.55.0

[PATCH v7 1/2] hung_task: Reset warning budget when problem gets resolved

[Aaron Tomlin]

From: Petr Mladek <pmladek@suse.com>

sysctl_hung_task_warnings counts how many hung tasks are reported.
The watchdog does not report anything once the limit is reached.
Currently, this budget is decremented permanently, meaning the kernel is
left blind to subsequent hung tasks even after the original issue resolves.

Keep the global sysctl_hung_task_warnings intact, and instead decrement
a copy (hung_task_warnings_printed) when warnings are printed. Reset
the copy back to the configured sysctl_hung_task_warnings limit once the
problem on the system gets resolved and check_hung_uninterruptible_tasks()
detects no hung tasks in a check interval.

Also keep the copy updated when the global sysctl_hung_task_warnings
value is updated via sysctl, and update documentation to reflect
the new behavior.

Assisted-by: gemini-3.5-flash
Suggested-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Tested-by: Lance Yang <lance.yang@linux.dev>
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 Documentation/admin-guide/sysctl/kernel.rst |  5 +--
 kernel/hung_task.c                          | 34 ++++++++++++++++-----
 2 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index c6994e55d141..b0f8e55efc1a 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -459,8 +459,9 @@ hung_task_warnings
 ==================
 
 The maximum number of warnings to report. During a check interval
-if a hung task is detected, this value is decreased by 1.
-When this value reaches 0, no more warnings will be reported.
+if a hung task is detected, the internal warning budget is decreased by 1.
+When this budget reaches 0, no more detailed warnings will be reported. The
+warning budget is reset to the configured limit when no hung task is found.
 This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.
 
 -1: report an infinite number of warnings.
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 6fcc94ce4ca9..4e1fb0db79d1 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -59,6 +59,8 @@ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs;
 
 static int __read_mostly sysctl_hung_task_warnings = 10;
 
+static int hung_task_warnings_printed = 10;
+
 static int __read_mostly did_panic;
 static bool hung_task_call_panic;
 
@@ -247,9 +249,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
 	 * accordingly
 	 */
-	if (sysctl_hung_task_warnings || hung_task_call_panic) {
-		if (sysctl_hung_task_warnings > 0)
-			sysctl_hung_task_warnings--;
+	if (hung_task_warnings_printed || hung_task_call_panic) {
+		if (hung_task_warnings_printed > 0)
+			hung_task_warnings_printed--;
 		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
 		       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
 		       (jiffies - t->last_switch_time) / HZ);
@@ -264,7 +266,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		sched_show_task(t);
 		debug_show_blocker(t, timeout);
 
-		if (!sysctl_hung_task_warnings)
+		if (!hung_task_warnings_printed)
 			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
 	}
 
@@ -304,7 +306,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	unsigned long last_break = jiffies;
 	struct task_struct *g, *t;
 	unsigned long this_round_count;
-	int need_warning = sysctl_hung_task_warnings;
+	int need_warning = hung_task_warnings_printed;
 	unsigned long si_mask = hung_task_si_mask;
 
 	/*
@@ -340,8 +342,10 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
  unlock:
 	rcu_read_unlock();
 
-	if (!this_round_count)
+	if (!this_round_count) {
+		hung_task_warnings_printed = sysctl_hung_task_warnings;
 		return;
+	}
 
 	if (need_warning || hung_task_call_panic) {
 		si_mask |= SYS_INFO_LOCKS;
@@ -425,6 +429,22 @@ static int proc_dohung_task_timeout_secs(const struct ctl_table *table, int writ
 	return ret;
 }
 
+static int proc_dohung_task_warnings(const struct ctl_table *table, int write,
+				     void *buffer,
+				     size_t *lenp, loff_t *ppos)
+{
+	int ret;
+
+	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
+
+	if (ret || !write)
+		return ret;
+
+	hung_task_warnings_printed = sysctl_hung_task_warnings;
+
+	return 0;
+}
+
 /*
  * This is needed for proc_doulongvec_minmax of sysctl_hung_task_timeout_secs
  * and hung_task_check_interval_secs
@@ -480,7 +500,7 @@ static const struct ctl_table hung_task_sysctls[] = {
 		.data		= &sysctl_hung_task_warnings,
 		.maxlen		= sizeof(int),
 		.mode		= 0644,
-		.proc_handler	= proc_dointvec_minmax,
+		.proc_handler	= proc_dohung_task_warnings,
 		.extra1		= SYSCTL_NEG_ONE,
 	},
 	{
-- 
2.55.0

Re: [PATCH v7 1/2] hung_task: Reset warning budget when problem gets resolved

[Lance Yang]

On Tue, Aug 04, 2026 at 11:54:05AM -0400, Aaron Tomlin wrote:
>From: Petr Mladek <pmladek@suse.com>
>
>sysctl_hung_task_warnings counts how many hung tasks are reported.
>The watchdog does not report anything once the limit is reached.
>Currently, this budget is decremented permanently, meaning the kernel is
>left blind to subsequent hung tasks even after the original issue resolves.
>
>Keep the global sysctl_hung_task_warnings intact, and instead decrement
>a copy (hung_task_warnings_printed) when warnings are printed. Reset
>the copy back to the configured sysctl_hung_task_warnings limit once the
>problem on the system gets resolved and check_hung_uninterruptible_tasks()
>detects no hung tasks in a check interval.
>
>Also keep the copy updated when the global sysctl_hung_task_warnings
>value is updated via sysctl, and update documentation to reflect
>the new behavior.

I'd go with something like:

"
hung_task: separate warning limit from runtime budget

sysctl_hung_task_warnings currently holds both the configured warning
limit and the remaining budget. Each detailed report decrements the
sysctl, so once it reaches zero, the configured limit is lost and cannot
be restored automatically.

Keep sysctl_hung_task_warnings unchanged and track the remaining budget
in hung_task_warnings_printed. Reset the runtime budget when a watchdog
check sees no hung tasks or when userspace writes a new sysctl value.
"

[...]

Cheers, Lance

Re: [PATCH v7 1/2] hung_task: Reset warning budget when problem gets resolved

[Aaron Tomlin]

On Wed, Aug 05, 2026 at 12:40:10AM +0800, Lance Yang wrote:
> 
> On Tue, Aug 04, 2026 at 11:54:05AM -0400, Aaron Tomlin wrote:
> >From: Petr Mladek <pmladek@suse.com>
> >
> >sysctl_hung_task_warnings counts how many hung tasks are reported.
> >The watchdog does not report anything once the limit is reached.
> >Currently, this budget is decremented permanently, meaning the kernel is
> >left blind to subsequent hung tasks even after the original issue resolves.
> >
> >Keep the global sysctl_hung_task_warnings intact, and instead decrement
> >a copy (hung_task_warnings_printed) when warnings are printed. Reset
> >the copy back to the configured sysctl_hung_task_warnings limit once the
> >problem on the system gets resolved and check_hung_uninterruptible_tasks()
> >detects no hung tasks in a check interval.
> >
> >Also keep the copy updated when the global sysctl_hung_task_warnings
> >value is updated via sysctl, and update documentation to reflect
> >the new behavior.
> 
> I'd go with something like:
> 
> "
> hung_task: separate warning limit from runtime budget
> 
> sysctl_hung_task_warnings currently holds both the configured warning
> limit and the remaining budget. Each detailed report decrements the
> sysctl, so once it reaches zero, the configured limit is lost and cannot
> be restored automatically.
> 
> Keep sysctl_hung_task_warnings unchanged and track the remaining budget
> in hung_task_warnings_printed. Reset the runtime budget when a watchdog
> check sees no hung tasks or when userspace writes a new sysctl value.
> "
> 
> [...]
> 
> Cheers, Lance

Acknowledged.

-- 
Aaron Tomlin

[PATCH v7 2/2] hung_task: Always print basic hung task info header

[Aaron Tomlin]

From: Petr Mladek <pmladek@suse.com>

Currently, hung_task_info() completely suppresses any warning messages
once the warning budget is reached. This leaves the system completely
blind to any future hung tasks, even though we only wanted to suppress
heavy process stack dumps.

Modify hung_task_info() to always print the basic single-line error message
about a hung task, regardless of whether the warning budget has been
exhausted. Restrict only the verbose details (such as stack dumps,
taint/release status, and blockers) to when the warning budget is intact
or panic is triggered.

Additionally, update the notice printed upon warning exhaustion to clarify
that future reports will only omit process details rather than being completely
suppressed.

Assisted-by: gemini-3.5-flash
Suggested-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 kernel/hung_task.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 4e1fb0db79d1..6ebb3a87ac65 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -244,17 +244,19 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		hung_task_call_panic = true;
 	}
 
+	/* Always print the blocked message */
+	pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
+	       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
+	       (jiffies - t->last_switch_time) / HZ);
+
 	/*
 	 * The given task did not get scheduled for more than
 	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
-	 * accordingly
+	 * accordingly with full details if the budget is not exhausted.
 	 */
 	if (hung_task_warnings_printed || hung_task_call_panic) {
 		if (hung_task_warnings_printed > 0)
 			hung_task_warnings_printed--;
-		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
-		       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
-		       (jiffies - t->last_switch_time) / HZ);
 		pr_err("      %s %s %.*s\n",
 			print_tainted(), init_utsname()->release,
 			(int)strcspn(init_utsname()->version, " "),
@@ -267,7 +269,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		debug_show_blocker(t, timeout);
 
 		if (!hung_task_warnings_printed)
-			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
+			pr_info("Future hung task reports won't print details about each process, see sysctl kernel.hung_task_warnings\n");
 	}
 
 	touch_nmi_watchdog();
-- 
2.55.0

Re: [PATCH v7 2/2] hung_task: Always print basic hung task info header

[Lance Yang]

On Tue, Aug 04, 2026 at 11:54:06AM -0400, Aaron Tomlin wrote:
>From: Petr Mladek <pmladek@suse.com>
>
>Currently, hung_task_info() completely suppresses any warning messages
>once the warning budget is reached. This leaves the system completely
>blind to any future hung tasks, even though we only wanted to suppress
>heavy process stack dumps.
>
>Modify hung_task_info() to always print the basic single-line error message
>about a hung task, regardless of whether the warning budget has been
>exhausted. Restrict only the verbose details (such as stack dumps,
>taint/release status, and blockers) to when the warning budget is intact
>or panic is triggered.
>
>Additionally, update the notice printed upon warning exhaustion to clarify
>that future reports will only omit process details rather than being completely
>suppressed.

I'd keep it simple and go with:

"
hung_task: always print the blocked-task line

Once the warning budget is exhausted, hung_task_info() stops printing any
per-task message. Detection still increments hung_task_detect_count and
emits trace_sched_process_hang(), but dmesg no longer shows which task is
blocked.

Move the "INFO: task ... blocked" line outside the budget check. Keep the
rest of the per-task report budgeted and update the exhaustion message
accordingly.
"

[...]

Cheers, Lance

Re: [PATCH v7 2/2] hung_task: Always print basic hung task info header

[Aaron Tomlin]

On Wed, Aug 05, 2026 at 12:42:51AM +0800, Lance Yang wrote:
> 
> On Tue, Aug 04, 2026 at 11:54:06AM -0400, Aaron Tomlin wrote:
> >From: Petr Mladek <pmladek@suse.com>
> >
> >Currently, hung_task_info() completely suppresses any warning messages
> >once the warning budget is reached. This leaves the system completely
> >blind to any future hung tasks, even though we only wanted to suppress
> >heavy process stack dumps.
> >
> >Modify hung_task_info() to always print the basic single-line error message
> >about a hung task, regardless of whether the warning budget has been
> >exhausted. Restrict only the verbose details (such as stack dumps,
> >taint/release status, and blockers) to when the warning budget is intact
> >or panic is triggered.
> >
> >Additionally, update the notice printed upon warning exhaustion to clarify
> >that future reports will only omit process details rather than being completely
> >suppressed.
> 
> I'd keep it simple and go with:
> 
> "
> hung_task: always print the blocked-task line
> 
> Once the warning budget is exhausted, hung_task_info() stops printing any
> per-task message. Detection still increments hung_task_detect_count and
> emits trace_sched_process_hang(), but dmesg no longer shows which task is
> blocked.
> 
> Move the "INFO: task ... blocked" line outside the budget check. Keep the
> rest of the per-task report budgeted and update the exhaustion message
> accordingly.
> "
> 
> [...]
> 
> Cheers, Lance

Acknowledged.

-- 
Aaron Tomlin

Re: [PATCH v7 0/2] hung_task: Improve warning budget handling and task reporting

[Lance Yang]

On Tue, Aug 04, 2026 at 11:54:04AM -0400, Aaron Tomlin wrote:
[...]

A couple of things look off ...

>Petr Mladek (2):
>  hung_task: Reset warning budget when problem gets resolved
>  hung_task: Always print basic hung task info header

The authorship is still wrong ... Petr is listed in the in-body From:,
but there is no matching Signed-off-by from him. That in-body From:
should be dropped :)

All patch changelogs read AI-ish: lots of words, but unclear on what
changes and why ...

Cheers, Lance

Re: [PATCH v7 0/2] hung_task: Improve warning budget handling and task reporting

[Aaron Tomlin]

On Wed, Aug 05, 2026 at 12:30:15AM +0800, Lance Yang wrote:
> 
> On Tue, Aug 04, 2026 at 11:54:04AM -0400, Aaron Tomlin wrote:
> [...]
> 
> A couple of things look off ...
> 
> >Petr Mladek (2):
> >  hung_task: Reset warning budget when problem gets resolved
> >  hung_task: Always print basic hung task info header
> 
> The authorship is still wrong ... Petr is listed in the in-body From:,
> but there is no matching Signed-off-by from him. That in-body From:
> should be dropped :)
> 
> All patch changelogs read AI-ish: lots of words, but unclear on what
> changes and why ...
> 
> Cheers, Lance

Hi Lance,

Sorry, I missed that.

I didn't describe the differences between v6 and v7.

-- 
Aaron Tomlin

Re: [PATCH v7 0/2] hung_task: Improve warning budget handling and task reporting

[Lance Yang]

On 2026/8/5 01:32, Aaron Tomlin wrote:
> On Wed, Aug 05, 2026 at 12:30:15AM +0800, Lance Yang wrote:
>>
>> On Tue, Aug 04, 2026 at 11:54:04AM -0400, Aaron Tomlin wrote:
>> [...]
>>
>> A couple of things look off ...
>>
>>> Petr Mladek (2):
>>>   hung_task: Reset warning budget when problem gets resolved
>>>   hung_task: Always print basic hung task info header
>>
>> The authorship is still wrong ... Petr is listed in the in-body From:,
>> but there is no matching Signed-off-by from him. That in-body From:
>> should be dropped :)
>>
>> All patch changelogs read AI-ish: lots of words, but unclear on what
>> changes and why ...
>>
>> Cheers, Lance
> 
> Hi Lance,
> 
> Sorry, I missed that.
> 
> I didn't describe the differences between v6 and v7.

Was mostly looking at code and didn't pay enough attention to
the wording. That's on me ...