Re: [PATCH v6] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[Petr Mladek <pmladek@suse.com>]

On Tue 2026-07-21 13:08:32, Lance Yang wrote:
> 
> On Sun, Jul 19, 2026 at 12:13:05PM -0400, Aaron Tomlin wrote:
> >Currently, during severe lock contention, multiple tasks can hang while
> >waiting on the exact same resource. The khungtaskd kthread
> >indiscriminately reports every single instance with a stack trace.
> >This can roll the kernel ring buffer and prematurely exhaust the
> >kernel.hung_task_warnings budget. Consequently, the kernel is left
> >entirely blind to subsequent, unrelated deadlocks.
> >
> >To preserve the warning budget and ring buffer without sacrificing
> >observability, this patch introduces a two-tier deduplication mechanism:
> >
> >    1. Introduces a hung_task_reported in task_struct. If a task remains
> >       hung across multiple check intervals, khungtaskd suppresses
> >       redundant stack traces for that specific task until it makes
> >       progress (verified via context switch counters). Furthermore, it
> >       is packed into an existing compiler alignment hole, consuming
> >       zero additional memory
> >
> >    2. For tasks blocked on a lock, we leverage the
> >       CONFIG_DETECT_HUNG_TASK_BLOCKER infrastructure. By tracking the
> >       exact memory address of the blocker lock in a persistent array of
> >       size 32, we deduplicate warning reports for tasks waiting on the
> >       same resource across check intervals. If the blocker is already
> >       tracked, the warning is suppressed and the warning budget is
> >       preserved
> >
> >    3. For duplicate tasks, we still print the single-line "INFO: task
> >       ..." message and trigger tracepoint trace_sched_process_hang().
> >       It merely skips calling sched_show_task() and
> >       debug_show_blocker(), printing a concise suppression notice
> >       instead
> >
> >    4. Once the hang resolves (i.e., no hung tasks are detected in a
> >       check round), the budget is restored to the value captured at the
> >       start of the hang, and the blocker array is cleared, accompanied
> >       by a recovery message in the ring buffer
> >
> >Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
> >---
> 
> Sorry for the late reply.

Don't worry. IMHO, a reply within one week is perfectly
fine. Sometimes even this is not possible.

> Still, v6 looks far too complicated ... and touches too much core code
> for the problem you're trying to address ...

I agree that it looks too complicated. I see several problems:

  1. Many changes are added in a single patch. I suggest to split
     it next time into more patches and add the features from
     the most important one, ...

  2. The code seems to be a bit over-engineered. There are too many
     variables hung_task_blockers_count, hung_task_has_active,
     warnings_decremented, skip_show_task, scan_completed,
     did_report, seen_blockers_mask, ... I believe that it might
     be done an easier way.

  3. Too much spaghetti code is added to the already large
     check_hung_uninterruptible_tasks() function. It might look
     better when the detection of the duplicated task is handled
     in a separate function, ...


> Looked at this again ... Petr's earlier breakdown[1] makes sense. Both
> problems are real:
> 
> "
> I would split this into two problems:
> 
> 1. A single lock contention might trigger hung_report for many tasks
>    waiting for the same lock. It bloats the kernel log and messages
>    might even get lost.
> 
> 2. The number of printed backtraces can be reduced by a global limit.
>    But the limit silences the hung task detector and system
>    administrators are blind once the limit is reached.
> "
> 
> IMO, that's the tradeoff: keep log noise down without leaving users
> blind for good once budget runs out ...
> 
> Still not sold on dedup, though. Sure, duplicate reports happen, but how
> often, really? Doesn't seem common enough to need this much code :)
> 
> If we really want to handle this in kernel, how about a simple mode
> switch for hung_task_warnings?
> 
>   0 = keep the current global budget
>   1 = allow up to hung_task_warnings reports in each scan

Honestly, I do not think that this is the right way to go. The more I think
about it, the more I believe that the global budget is simply wrong.

IMHO, we should do the following:

1. Reset the global budget when the situation is resolved and
   there is no hung task any longer.

2. The global budget should limit only printing all the details,
   especially backtraces. We should always print at least
   the task name, ...

3. We could prevent printing the same process again and again
   by adding "hung_task_reported" into struct task_struct.

   But then we should print some summary about how many hung tasks
   were found in the last round at least. So that we know that
   the problem persists.

   This should still be rather easy. But I am not sure if it is
   worth it.

4. We could add the filtering of the duplicated backtraces. But
   it has to be handled by some helper function.

   It looks like to most complicated part. I would personally skip
   it for now. We could always add it later when the above proposed
   changes are not enough to tune the kernel.hung_task_warnings
   value in the real life.


I tried to implement the 1st and 2nd change using gemini just to
see how it would look. And it looks good to me:

From 4843170a1317f2fd3bebc1efc63c387840a4ed0d Mon Sep 17 00:00:00 2001
From: Petr Mladek <pmladek@suse.com>
Date: Tue, 21 Jul 2026 14:09:33 +0200
Subject: [PATCH 1/2] hung_task: Reset warning budget when problem gets
 resolved

sysctl_hung_task_warnings counts how many hung tasks are reported.
The watchdog does not report anything once the limit is reached.
Currently, this budget is decremented permanently, meaning the kernel is
left blind to subsequent hung tasks even after the original issue resolves.

Keep the global sysctl_hung_task_warnings intact, and instead decrement
a copy (hung_task_warnings_printed) when warnings are printed. Reset
the copy back to the configured sysctl_hung_task_warnings limit once the
problem on the system gets resolved and check_hung_uninterruptible_tasks()
detects no hung tasks in a check interval.

Also keep the copy updated when the global sysctl_hung_task_warnings
value is updated via sysctl, and update documentation to reflect
the new behavior.

Assisted-by: gemini-3.5-flash
Signed-off-by: Petr Mladek <pmladek@suse.com>
---
 Documentation/admin-guide/sysctl/kernel.rst |  6 ++--
 kernel/hung_task.c                          | 34 ++++++++++++++++-----
 2 files changed, 31 insertions(+), 9 deletions(-)

diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index c6994e55d141..eaa1d837c739 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -459,8 +459,10 @@ hung_task_warnings
 ==================
 
 The maximum number of warnings to report. During a check interval
-if a hung task is detected, this value is decreased by 1.
-When this value reaches 0, no more warnings will be reported.
+if a hung task is detected, the internal warning budget is decreased by 1.
+When this budget reaches 0, no more warnings will be reported.
+The warning budget is reset back to the configured limit when the problem on the
+system gets resolved and no hung task is found.
 This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.
 
 -1: report an infinite number of warnings.
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 6fcc94ce4ca9..4e1fb0db79d1 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -59,6 +59,8 @@ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs;
 
 static int __read_mostly sysctl_hung_task_warnings = 10;
 
+static int hung_task_warnings_printed = 10;
+
 static int __read_mostly did_panic;
 static bool hung_task_call_panic;
 
@@ -247,9 +249,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
 	 * accordingly
 	 */
-	if (sysctl_hung_task_warnings || hung_task_call_panic) {
-		if (sysctl_hung_task_warnings > 0)
-			sysctl_hung_task_warnings--;
+	if (hung_task_warnings_printed || hung_task_call_panic) {
+		if (hung_task_warnings_printed > 0)
+			hung_task_warnings_printed--;
 		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
 		       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
 		       (jiffies - t->last_switch_time) / HZ);
@@ -264,7 +266,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		sched_show_task(t);
 		debug_show_blocker(t, timeout);
 
-		if (!sysctl_hung_task_warnings)
+		if (!hung_task_warnings_printed)
 			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
 	}
 
@@ -304,7 +306,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	unsigned long last_break = jiffies;
 	struct task_struct *g, *t;
 	unsigned long this_round_count;
-	int need_warning = sysctl_hung_task_warnings;
+	int need_warning = hung_task_warnings_printed;
 	unsigned long si_mask = hung_task_si_mask;
 
 	/*
@@ -340,8 +342,10 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
  unlock:
 	rcu_read_unlock();
 
-	if (!this_round_count)
+	if (!this_round_count) {
+		hung_task_warnings_printed = sysctl_hung_task_warnings;
 		return;
+	}
 
 	if (need_warning || hung_task_call_panic) {
 		si_mask |= SYS_INFO_LOCKS;
@@ -425,6 +429,22 @@ static int proc_dohung_task_timeout_secs(const struct ctl_table *table, int writ
 	return ret;
 }
 
+static int proc_dohung_task_warnings(const struct ctl_table *table, int write,
+				     void *buffer,
+				     size_t *lenp, loff_t *ppos)
+{
+	int ret;
+
+	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
+
+	if (ret || !write)
+		return ret;
+
+	hung_task_warnings_printed = sysctl_hung_task_warnings;
+
+	return 0;
+}
+
 /*
  * This is needed for proc_doulongvec_minmax of sysctl_hung_task_timeout_secs
  * and hung_task_check_interval_secs
@@ -480,7 +500,7 @@ static const struct ctl_table hung_task_sysctls[] = {
 		.data		= &sysctl_hung_task_warnings,
 		.maxlen		= sizeof(int),
 		.mode		= 0644,
-		.proc_handler	= proc_dointvec_minmax,
+		.proc_handler	= proc_dohung_task_warnings,
 		.extra1		= SYSCTL_NEG_ONE,
 	},
 	{
-- 
2.55.0



From 50f69511d706484d9378c019f8cbaa3232555fa6 Mon Sep 17 00:00:00 2001
From: Petr Mladek <pmladek@suse.com>
Date: Tue, 21 Jul 2026 14:49:16 +0200
Subject: [PATCH 2/2] hung_task: Always print basic hung task info header

Currently, hung_task_info() completely suppresses any warning messages
once the warning budget is reached. This leaves the system completely
blind to any future hung tasks, even though we only wanted to suppress
heavy process stack dumps.

Modify hung_task_info() to always print the basic single-line error message
about a hung task, regardless of whether the warning budget has been
exhausted. Restrict only the verbose details (such as stack dumps,
taint/release status, and blockers) to when the warning budget is intact
or panic is triggered.

Additionally, update the notice printed upon warning exhaustion to clarify
that future reports will only omit process details rather than being completely
suppressed.

Assisted-by: gemini-3.5-flash
Signed-off-by: Petr Mladek <pmladek@suse.com>
---
 kernel/hung_task.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 4e1fb0db79d1..6ebb3a87ac65 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -244,17 +244,19 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		hung_task_call_panic = true;
 	}
 
+	/* Always print the blocked message */
+	pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
+	       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
+	       (jiffies - t->last_switch_time) / HZ);
+
 	/*
 	 * The given task did not get scheduled for more than
 	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
-	 * accordingly
+	 * accordingly with full details if the budget is not exhausted.
 	 */
 	if (hung_task_warnings_printed || hung_task_call_panic) {
 		if (hung_task_warnings_printed > 0)
 			hung_task_warnings_printed--;
-		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
-		       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
-		       (jiffies - t->last_switch_time) / HZ);
 		pr_err("      %s %s %.*s\n",
 			print_tainted(), init_utsname()->release,
 			(int)strcspn(init_utsname()->version, " "),
@@ -267,7 +269,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		debug_show_blocker(t, timeout);
 
 		if (!hung_task_warnings_printed)
-			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
+			pr_info("Future hung task reports won't print details about each process, see sysctl kernel.hung_task_warnings\n");
 	}
 
 	touch_nmi_watchdog();
-- 
2.55.0