[PATCH v6] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[PATCH v6] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[Aaron Tomlin]

Currently, during severe lock contention, multiple tasks can hang while
waiting on the exact same resource. The khungtaskd kthread
indiscriminately reports every single instance with a stack trace.
This can roll the kernel ring buffer and prematurely exhaust the
kernel.hung_task_warnings budget. Consequently, the kernel is left
entirely blind to subsequent, unrelated deadlocks.

To preserve the warning budget and ring buffer without sacrificing
observability, this patch introduces a two-tier deduplication mechanism:

    1. Introduces a hung_task_reported in task_struct. If a task remains
       hung across multiple check intervals, khungtaskd suppresses
       redundant stack traces for that specific task until it makes
       progress (verified via context switch counters). Furthermore, it
       is packed into an existing compiler alignment hole, consuming
       zero additional memory

    2. For tasks blocked on a lock, we leverage the
       CONFIG_DETECT_HUNG_TASK_BLOCKER infrastructure. By tracking the
       exact memory address of the blocker lock in a persistent array of
       size 32, we deduplicate warning reports for tasks waiting on the
       same resource across check intervals. If the blocker is already
       tracked, the warning is suppressed and the warning budget is
       preserved

    3. For duplicate tasks, we still print the single-line "INFO: task
       ..." message and trigger tracepoint trace_sched_process_hang().
       It merely skips calling sched_show_task() and
       debug_show_blocker(), printing a concise suppression notice
       instead

    4. Once the hang resolves (i.e., no hung tasks are detected in a
       check round), the budget is restored to the value captured at the
       start of the hang, and the blocker array is cleared, accompanied
       by a recovery message in the ring buffer

Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
Changes since v5:

 - Skipped hung_task_info() and sys_info() for tasks already reported in
   previous rounds to avoid log spam

 - Linked to v5: https://lore.kernel.org/lkml/20260712202100.123934-1-atomlin@atomlin.com/

Changes since v4:

 - Replaced the stack-local hashmap implementation with a persistent
   array (Petr Mladek)

 - Persistently track blocker addresses across scan intervals. Suppress
   warning reports and keep the sysctl_hung_task_warnings budget intact
   if the blocker is already tracked in the array (Petr Mladek)

 - Reset the warnings budget and clear the blocker array when the hang
   resolves (Petr Mladek)

 - Output a recovery message to the kernel ring buffer upon hang
   resolution

 - Linked to v4: https://lore.kernel.org/lkml/20260627205733.90983-1-atomlin@atomlin.com/

Changes since v3:

 - Deduct from the global budget if printing a full stack trace

 - Pivoted from heuristic Wait Channel hashing to deterministic
   blocker address hashing via CONFIG_DETECT_HUNG_TASK_BLOCKER

 - Replaced the hung_task_reported bit-field with a standalone u8 byte.
   Move hung_task_reported into an existing structural alignment hole
   within task_struct following blocked_lock, resulting in zero overall
   memory footprint increase and optimal cacheline grouping

 - Linked to v3: https://lore.kernel.org/lkml/20260621213756.43225-1-atomlin@atomlin.com/

Changes since v2:

 - Replaced the per-round cache flush with a task_struct bit-field for
   persistent cross-scan tracking, mitigating delayed budget exhaustion

 - Abandoned exact-stack hashing in favour of Wait Channel hashing

 - Transitioned from jhash() to hash_long() to optimise single-pointer
   hashing, and relocated the hash map to the local stack

 - Linked to v2: https://lore.kernel.org/lkml/20260620013559.1537893-1-atomlin@atomlin.com/

Changes since v1:

 - Preserve "INFO:" headers for all hung tasks; suppress only the stack
   dumps for duplicates (Masami Hiramatsu)

 - Print a clear notification when a trace is explicitly suppressed

 - Add #ifdef CONFIG_STACKTRACE guards to prevent Kconfig build errors

 - Optimise overhead by unwinding the stack only if a warning is
   actually going to be printed

 - Linked to v1: https://lore.kernel.org/lkml/20260617184841.1447955-1-atomlin@atomlin.com/
---
 include/linux/hung_task.h |   1 +
 include/linux/sched.h     |   3 +
 kernel/fork.c             |   1 +
 kernel/hung_task.c        | 126 +++++++++++++++++++++++++++++++++++---
 4 files changed, 124 insertions(+), 7 deletions(-)

diff --git a/include/linux/hung_task.h b/include/linux/hung_task.h
index c4403eeb7144..b19220613402 100644
--- a/include/linux/hung_task.h
+++ b/include/linux/hung_task.h
@@ -37,6 +37,7 @@
 
 #define BLOCKER_TYPE_MASK		0x03UL
 
+
 #ifdef CONFIG_DETECT_HUNG_TASK_BLOCKER
 static inline void hung_task_set_blocker(void *lock, unsigned long type)
 {
diff --git a/include/linux/sched.h b/include/linux/sched.h
index b3204a15d512..deb092ff8a9a 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1252,6 +1252,9 @@ struct task_struct {
 	struct mutex			*blocked_on;	/* lock we're blocked on */
 	raw_spinlock_t			blocked_lock;
 
+#ifdef CONFIG_DETECT_HUNG_TASK
+	u8				hung_task_reported;
+#endif
 	/*
 	 * The task that is boosting this task; a back link for the current
 	 * donor stack. Set in schedule() -> find_proxy_task() and only stable
diff --git a/kernel/fork.c b/kernel/fork.c
index 892a95214c54..8a5c08cc42d5 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -1569,6 +1569,7 @@ static int copy_mm(u64 clone_flags, struct task_struct *tsk)
 #ifdef CONFIG_DETECT_HUNG_TASK
 	tsk->last_switch_count = tsk->nvcsw + tsk->nivcsw;
 	tsk->last_switch_time = 0;
+	tsk->hung_task_reported = 0;
 #endif
 
 	tsk->mm = NULL;
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 6fcc94ce4ca9..6bb2b26693a1 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -25,6 +25,8 @@
 #include <linux/hung_task.h>
 #include <linux/rwsem.h>
 #include <linux/sys_info.h>
+#include <linux/hash.h>
+#include <linux/array_size.h>
 
 #include <trace/events/sched.h>
 
@@ -59,6 +61,16 @@ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs;
 
 static int __read_mostly sysctl_hung_task_warnings = 10;
 
+#ifdef CONFIG_DETECT_HUNG_TASK_BLOCKER
+#define HUNG_TASK_BLOCKERS_MAX 32
+static unsigned long hung_task_blockers[HUNG_TASK_BLOCKERS_MAX];
+static int hung_task_blockers_count;
+static bool hung_task_has_active;
+static int warnings_decremented;
+static_assert(HUNG_TASK_BLOCKERS_MAX <= BITS_PER_LONG,
+	      "HUNG_TASK_BLOCKERS_MAX exceeds BITS_PER_LONG");
+#endif
+
 static int __read_mostly did_panic;
 static bool hung_task_call_panic;
 
@@ -125,6 +137,7 @@ static bool task_is_hung(struct task_struct *t, unsigned long timeout)
 	if (switch_count != t->last_switch_count) {
 		t->last_switch_count = switch_count;
 		t->last_switch_time = jiffies;
+		t->hung_task_reported = 0;
 		return false;
 	}
 	if (time_is_after_jiffies(t->last_switch_time + timeout * HZ))
@@ -228,12 +241,14 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti
  * @t: Pointer to the detected hung task.
  * @timeout: Timeout threshold for detecting hung tasks
  * @this_round_count: Count of hung tasks detected in the current iteration
+ * @skip_show_task: Indicating if stack trace should be skipped
  *
  * Print structured information about the specified hung task, if warnings
  * are enabled or if the panic batch threshold is exceeded.
  */
 static void hung_task_info(struct task_struct *t, unsigned long timeout,
-			   unsigned long this_round_count)
+			   unsigned long this_round_count,
+			   unsigned int skip_show_task)
 {
 	trace_sched_process_hang(t);
 
@@ -248,8 +263,16 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 	 * accordingly
 	 */
 	if (sysctl_hung_task_warnings || hung_task_call_panic) {
-		if (sysctl_hung_task_warnings > 0)
+		/*
+		 * Do not exhaust the global warning budget for duplicates;
+		 * only decrement if a full stack trace is being printed.
+		 */
+		if (!skip_show_task && sysctl_hung_task_warnings > 0) {
 			sysctl_hung_task_warnings--;
+#ifdef CONFIG_DETECT_HUNG_TASK_BLOCKER
+			warnings_decremented++;
+#endif
+		}
 		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
 		       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
 		       (jiffies - t->last_switch_time) / HZ);
@@ -261,8 +284,12 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 			pr_err("      Blocked by coredump.\n");
 		pr_err("\"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\""
 			" disables this message.\n");
-		sched_show_task(t);
-		debug_show_blocker(t, timeout);
+		if (!skip_show_task) {
+			sched_show_task(t);
+			debug_show_blocker(t, timeout);
+		} else {
+			pr_err("      Stack trace suppressed. Already reported or duplicate\n");
+		}
 
 		if (!sysctl_hung_task_warnings)
 			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
@@ -306,6 +333,14 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	unsigned long this_round_count;
 	int need_warning = sysctl_hung_task_warnings;
 	unsigned long si_mask = hung_task_si_mask;
+	unsigned int skip_show_task;
+	bool scan_completed = false;
+	bool did_report = false;
+#ifdef CONFIG_DETECT_HUNG_TASK_BLOCKER
+	unsigned long blocker;
+	static int warnings_reset_value;
+	unsigned long seen_blockers_mask = 0;
+#endif
 
 	/*
 	 * If the system crashed already then all bets are off,
@@ -326,6 +361,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 		}
 
 		if (task_is_hung(t, timeout)) {
+			skip_show_task = t->hung_task_reported;
 			/*
 			 * Increment the global counter so that userspace could
 			 * start migrating tasks ASAP. But count the current
@@ -334,16 +370,92 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 			 */
 			atomic_long_inc(&sysctl_hung_task_detect_count);
 			this_round_count++;
-			hung_task_info(t, timeout, this_round_count);
+
+#ifdef CONFIG_DETECT_HUNG_TASK_BLOCKER
+			if (!hung_task_has_active) {
+				warnings_reset_value = sysctl_hung_task_warnings;
+				warnings_decremented = 0;
+				hung_task_has_active = true;
+			}
+			blocker = READ_ONCE(t->blocker);
+			if (blocker) {
+				bool found = false;
+				int i, blocker_idx = -1;
+
+				blocker &= ~BLOCKER_TYPE_MASK;
+				for (i = 0; i < hung_task_blockers_count; i++) {
+					if (hung_task_blockers[i] == blocker) {
+						found = true;
+						blocker_idx = i;
+						break;
+					}
+				}
+				if (found || t->hung_task_reported) {
+					skip_show_task = 1;
+					if (found)
+						seen_blockers_mask |= (1UL << blocker_idx);
+				} else {
+					skip_show_task = 0;
+					if (hung_task_blockers_count < ARRAY_SIZE(hung_task_blockers)) {
+						blocker_idx = hung_task_blockers_count;
+						hung_task_blockers[hung_task_blockers_count++] = blocker;
+						seen_blockers_mask |= (1UL << blocker_idx);
+					} else {
+						pr_warn_once("INFO: tracked blocker array full. Untracked blockers may trigger duplicate warnings\n");
+					}
+				}
+			}
+#endif
+
+			if (!t->hung_task_reported) {
+				hung_task_info(t, timeout, this_round_count,
+					       skip_show_task);
+				t->hung_task_reported = 1;
+				if (!skip_show_task)
+					did_report = true;
+			}
 		}
 	}
+	scan_completed = true;
  unlock:
 	rcu_read_unlock();
 
-	if (!this_round_count)
+	if (!this_round_count) {
+#ifdef CONFIG_DETECT_HUNG_TASK_BLOCKER
+		if (scan_completed && hung_task_has_active) {
+			if (sysctl_hung_task_warnings == warnings_reset_value - warnings_decremented) {
+				pr_info("INFO: hung tasks resolved. Cleared %d tracked blocker(s). Warning budget restored to %d\n",
+					hung_task_blockers_count, warnings_reset_value);
+				sysctl_hung_task_warnings = warnings_reset_value;
+			} else {
+				pr_info("INFO: hung tasks resolved. Cleared %d tracked blocker(s). Warning budget manual change retained at %d\n",
+					hung_task_blockers_count, sysctl_hung_task_warnings);
+			}
+			for (int i = 0; i < ARRAY_SIZE(hung_task_blockers); i++)
+				hung_task_blockers[i] = 0;
+			hung_task_blockers_count = 0;
+			hung_task_has_active = false;
+		}
+#endif
 		return;
+	}
+
+#ifdef CONFIG_DETECT_HUNG_TASK_BLOCKER
+	if (scan_completed) {
+		int i, write_idx = 0;
+		for (i = 0; i < hung_task_blockers_count; i++) {
+			if (seen_blockers_mask & (1UL << i)) {
+				hung_task_blockers[write_idx++] = hung_task_blockers[i];
+			}
+		}
+		for (i = write_idx; i < hung_task_blockers_count; i++) {
+			hung_task_blockers[i] = 0;
+		}
+		hung_task_blockers_count = write_idx;
+	}
+#endif
 
-	if (need_warning || hung_task_call_panic) {
+	if ((need_warning && did_report) || hung_task_call_panic) {
 		si_mask |= SYS_INFO_LOCKS;
 
 		if (sysctl_hung_task_all_cpu_backtrace)
-- 
2.54.0

Re: [PATCH v6] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[Andrew Morton]

On Sun, 19 Jul 2026 12:13:05 -0400 Aaron Tomlin <atomlin@atomlin.com> wrote:

> Currently, during severe lock contention, multiple tasks can hang while
> waiting on the exact same resource. The khungtaskd kthread
> indiscriminately reports every single instance with a stack trace.
> This can roll the kernel ring buffer and prematurely exhaust the
> kernel.hung_task_warnings budget. Consequently, the kernel is left
> entirely blind to subsequent, unrelated deadlocks.

This does seem something we'd like to address.

> To preserve the warning budget and ring buffer without sacrificing
> observability, this patch introduces a two-tier deduplication mechanism:
> 

Thanks.  Hopefully Petr and Lance will be able to review this version.

AI review might have found an issue:
	https://sashiko.dev/#/patchset/20260719161305.428947-1-atomlin@atomlin.com

Re: [PATCH v6] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[Aaron Tomlin]

On Sun, Jul 19, 2026 at 11:02:57PM -0700, Andrew Morton wrote:
> On Sun, 19 Jul 2026 12:13:05 -0400 Aaron Tomlin <atomlin@atomlin.com> wrote:
> 
> > Currently, during severe lock contention, multiple tasks can hang while
> > waiting on the exact same resource. The khungtaskd kthread
> > indiscriminately reports every single instance with a stack trace.
> > This can roll the kernel ring buffer and prematurely exhaust the
> > kernel.hung_task_warnings budget. Consequently, the kernel is left
> > entirely blind to subsequent, unrelated deadlocks.
> 
> This does seem something we'd like to address.
> 
> > To preserve the warning budget and ring buffer without sacrificing
> > observability, this patch introduces a two-tier deduplication mechanism:
> > 
> 
> Thanks.  Hopefully Petr and Lance will be able to review this version.
> 
> AI review might have found an issue:
> 	https://sashiko.dev/#/patchset/20260719161305.428947-1-atomlin@atomlin.com

Hi Andrew,

Thank you for you feedback.

This is indeed a legitimate issue.

I have moved the panic threshold check out of hung_task_info() and placing
it directly into check_hung_uninterruptible_tasks() right after
this_round_count is incremented.

This ensures that:

    1.  The panic threshold is correctly evaluated for all hung tasks,
        regardless of whether they have already been reported

    2.  If the panic threshold is reached, hung_task_call_panic is set to
        true, ensuring the watchdog successfully panics and dumps the
        system state

    3.  We still completely suppress the redundant stack traces for
        already-reported tasks (i.e., preventing log spam and preserving
        the global warning budget)


Kind regards,
-- 
Aaron Tomlin

Re: [PATCH v6] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[Lance Yang]

On Sun, Jul 19, 2026 at 12:13:05PM -0400, Aaron Tomlin wrote:
>Currently, during severe lock contention, multiple tasks can hang while
>waiting on the exact same resource. The khungtaskd kthread
>indiscriminately reports every single instance with a stack trace.
>This can roll the kernel ring buffer and prematurely exhaust the
>kernel.hung_task_warnings budget. Consequently, the kernel is left
>entirely blind to subsequent, unrelated deadlocks.
>
>To preserve the warning budget and ring buffer without sacrificing
>observability, this patch introduces a two-tier deduplication mechanism:
>
>    1. Introduces a hung_task_reported in task_struct. If a task remains
>       hung across multiple check intervals, khungtaskd suppresses
>       redundant stack traces for that specific task until it makes
>       progress (verified via context switch counters). Furthermore, it
>       is packed into an existing compiler alignment hole, consuming
>       zero additional memory
>
>    2. For tasks blocked on a lock, we leverage the
>       CONFIG_DETECT_HUNG_TASK_BLOCKER infrastructure. By tracking the
>       exact memory address of the blocker lock in a persistent array of
>       size 32, we deduplicate warning reports for tasks waiting on the
>       same resource across check intervals. If the blocker is already
>       tracked, the warning is suppressed and the warning budget is
>       preserved
>
>    3. For duplicate tasks, we still print the single-line "INFO: task
>       ..." message and trigger tracepoint trace_sched_process_hang().
>       It merely skips calling sched_show_task() and
>       debug_show_blocker(), printing a concise suppression notice
>       instead
>
>    4. Once the hang resolves (i.e., no hung tasks are detected in a
>       check round), the budget is restored to the value captured at the
>       start of the hang, and the blocker array is cleared, accompanied
>       by a recovery message in the ring buffer
>
>Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
>---

Sorry for the late reply.

Still, v6 looks far too complicated ... and touches too much core code
for the problem you're trying to address ...

Looked at this again ... Petr's earlier breakdown[1] makes sense. Both
problems are real:

"
I would split this into two problems:

1. A single lock contention might trigger hung_report for many tasks
   waiting for the same lock. It bloats the kernel log and messages
   might even get lost.

2. The number of printed backtraces can be reduced by a global limit.
   But the limit silences the hung task detector and system
   administrators are blind once the limit is reached.
"

IMO, that's the tradeoff: keep log noise down without leaving users
blind for good once budget runs out ...

Still not sold on dedup, though. Sure, duplicate reports happen, but how
often, really? Doesn't seem common enough to need this much code :)

If we really want to handle this in kernel, how about a simple mode
switch for hung_task_warnings?

  0 = keep the current global budget
  1 = allow up to hung_task_warnings reports in each scan

---8<---
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 6fcc94ce4ca9..4ec2937a98e6 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -58,6 +58,7 @@ unsigned long __read_mostly sysctl_hung_task_timeout_secs = CONFIG_DEFAULT_HUNG_
 static unsigned long __read_mostly sysctl_hung_task_check_interval_secs;

 static int __read_mostly sysctl_hung_task_warnings = 10;
+static int __read_mostly sysctl_hung_task_warnings_mode;

 static int __read_mostly did_panic;
 static bool hung_task_call_panic;
@@ -228,12 +229,13 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti
  * @t: Pointer to the detected hung task.
  * @timeout: Timeout threshold for detecting hung tasks
  * @this_round_count: Count of hung tasks detected in the current iteration
+ * @warning_budget: Warning budget for reporting hung tasks
  *
  * Print structured information about the specified hung task, if warnings
  * are enabled or if the panic batch threshold is exceeded.
  */
 static void hung_task_info(struct task_struct *t, unsigned long timeout,
-			   unsigned long this_round_count)
+			   unsigned long this_round_count, int *warning_budget)
 {
 	trace_sched_process_hang(t);

@@ -247,9 +249,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
 	 * accordingly
 	 */
-	if (sysctl_hung_task_warnings || hung_task_call_panic) {
-		if (sysctl_hung_task_warnings > 0)
-			sysctl_hung_task_warnings--;
+	if (*warning_budget || hung_task_call_panic) {
+		if (*warning_budget > 0)
+			(*warning_budget)--;
 		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
 		       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
 		       (jiffies - t->last_switch_time) / HZ);
@@ -264,8 +266,8 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		sched_show_task(t);
 		debug_show_blocker(t, timeout);

-		if (!sysctl_hung_task_warnings)
-			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
+		if (!*warning_budget)
+			pr_info("Hung task warning budget exhausted, see sysctl kernel.hung_task_warnings\n");
 	}

 	touch_nmi_watchdog();
@@ -305,6 +307,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	struct task_struct *g, *t;
 	unsigned long this_round_count;
 	int need_warning = sysctl_hung_task_warnings;
+	int warnings_left = need_warning;
+	int *warning_budget = sysctl_hung_task_warnings_mode ?
+			      &warnings_left : &sysctl_hung_task_warnings;
 	unsigned long si_mask = hung_task_si_mask;

 	/*
@@ -334,7 +339,8 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 			 */
 			atomic_long_inc(&sysctl_hung_task_detect_count);
 			this_round_count++;
-			hung_task_info(t, timeout, this_round_count);
+			hung_task_info(t, timeout, this_round_count,
+				       warning_budget);
 		}
 	}
  unlock:
@@ -483,6 +489,15 @@ static const struct ctl_table hung_task_sysctls[] = {
 		.proc_handler	= proc_dointvec_minmax,
 		.extra1		= SYSCTL_NEG_ONE,
 	},
+	{
+		.procname	= "hung_task_warnings_mode",
+		.data		= &sysctl_hung_task_warnings_mode,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= SYSCTL_ZERO,
+		.extra2		= SYSCTL_ONE,
+	},
 	{
 		.procname	= "hung_task_detect_count",
 		.maxlen		= sizeof(unsigned long),
---

Per-scan mode avoids the permanent blind spot, while still letting users
pick whichever behavior fits their setup.

Keeps the change local, too. If that turns out not to be enough, sure,
revisit dedup then. That's the direction I'd support :D

[1] https://lore.kernel.org/lkml/akZlVgbImH6Jqy55@pathway.suse.cz/

Thanks, Lance

Re: [PATCH v6] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[Aaron Tomlin]

On Tue, Jul 21, 2026 at 01:08:32PM +0800, Lance Yang wrote:
> 
> On Sun, Jul 19, 2026 at 12:13:05PM -0400, Aaron Tomlin wrote:
> >Currently, during severe lock contention, multiple tasks can hang while
> >waiting on the exact same resource. The khungtaskd kthread
> >indiscriminately reports every single instance with a stack trace.
> >This can roll the kernel ring buffer and prematurely exhaust the
> >kernel.hung_task_warnings budget. Consequently, the kernel is left
> >entirely blind to subsequent, unrelated deadlocks.
> >
> >To preserve the warning budget and ring buffer without sacrificing
> >observability, this patch introduces a two-tier deduplication mechanism:
> >
> >    1. Introduces a hung_task_reported in task_struct. If a task remains
> >       hung across multiple check intervals, khungtaskd suppresses
> >       redundant stack traces for that specific task until it makes
> >       progress (verified via context switch counters). Furthermore, it
> >       is packed into an existing compiler alignment hole, consuming
> >       zero additional memory
> >
> >    2. For tasks blocked on a lock, we leverage the
> >       CONFIG_DETECT_HUNG_TASK_BLOCKER infrastructure. By tracking the
> >       exact memory address of the blocker lock in a persistent array of
> >       size 32, we deduplicate warning reports for tasks waiting on the
> >       same resource across check intervals. If the blocker is already
> >       tracked, the warning is suppressed and the warning budget is
> >       preserved
> >
> >    3. For duplicate tasks, we still print the single-line "INFO: task
> >       ..." message and trigger tracepoint trace_sched_process_hang().
> >       It merely skips calling sched_show_task() and
> >       debug_show_blocker(), printing a concise suppression notice
> >       instead
> >
> >    4. Once the hang resolves (i.e., no hung tasks are detected in a
> >       check round), the budget is restored to the value captured at the
> >       start of the hang, and the blocker array is cleared, accompanied
> >       by a recovery message in the ring buffer
> >
> >Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
> >---
> 
> Sorry for the late reply.
> 
> Still, v6 looks far too complicated ... and touches too much core code
> for the problem you're trying to address ...
> 
> Looked at this again ... Petr's earlier breakdown[1] makes sense. Both
> problems are real:
> 
> "
> I would split this into two problems:
> 
> 1. A single lock contention might trigger hung_report for many tasks
>    waiting for the same lock. It bloats the kernel log and messages
>    might even get lost.
> 
> 2. The number of printed backtraces can be reduced by a global limit.
>    But the limit silences the hung task detector and system
>    administrators are blind once the limit is reached.
> "
> 
> IMO, that's the tradeoff: keep log noise down without leaving users
> blind for good once budget runs out ...
> 
> Still not sold on dedup, though. Sure, duplicate reports happen, but how
> often, really? Doesn't seem common enough to need this much code :)
> 
> If we really want to handle this in kernel, how about a simple mode
> switch for hung_task_warnings?
> 
>   0 = keep the current global budget
>   1 = allow up to hung_task_warnings reports in each scan
> 
> ---8<---
> diff --git a/kernel/hung_task.c b/kernel/hung_task.c
> index 6fcc94ce4ca9..4ec2937a98e6 100644
> --- a/kernel/hung_task.c
> +++ b/kernel/hung_task.c
> @@ -58,6 +58,7 @@ unsigned long __read_mostly sysctl_hung_task_timeout_secs = CONFIG_DEFAULT_HUNG_
>  static unsigned long __read_mostly sysctl_hung_task_check_interval_secs;
> 
>  static int __read_mostly sysctl_hung_task_warnings = 10;
> +static int __read_mostly sysctl_hung_task_warnings_mode;
> 
>  static int __read_mostly did_panic;
>  static bool hung_task_call_panic;
> @@ -228,12 +229,13 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti
>   * @t: Pointer to the detected hung task.
>   * @timeout: Timeout threshold for detecting hung tasks
>   * @this_round_count: Count of hung tasks detected in the current iteration
> + * @warning_budget: Warning budget for reporting hung tasks
>   *
>   * Print structured information about the specified hung task, if warnings
>   * are enabled or if the panic batch threshold is exceeded.
>   */
>  static void hung_task_info(struct task_struct *t, unsigned long timeout,
> -			   unsigned long this_round_count)
> +			   unsigned long this_round_count, int *warning_budget)
>  {
>  	trace_sched_process_hang(t);
> 
> @@ -247,9 +249,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
>  	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
>  	 * accordingly
>  	 */
> -	if (sysctl_hung_task_warnings || hung_task_call_panic) {
> -		if (sysctl_hung_task_warnings > 0)
> -			sysctl_hung_task_warnings--;
> +	if (*warning_budget || hung_task_call_panic) {
> +		if (*warning_budget > 0)
> +			(*warning_budget)--;
>  		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
>  		       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
>  		       (jiffies - t->last_switch_time) / HZ);
> @@ -264,8 +266,8 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
>  		sched_show_task(t);
>  		debug_show_blocker(t, timeout);
> 
> -		if (!sysctl_hung_task_warnings)
> -			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
> +		if (!*warning_budget)
> +			pr_info("Hung task warning budget exhausted, see sysctl kernel.hung_task_warnings\n");
>  	}
> 
>  	touch_nmi_watchdog();
> @@ -305,6 +307,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
>  	struct task_struct *g, *t;
>  	unsigned long this_round_count;
>  	int need_warning = sysctl_hung_task_warnings;
> +	int warnings_left = need_warning;
> +	int *warning_budget = sysctl_hung_task_warnings_mode ?
> +			      &warnings_left : &sysctl_hung_task_warnings;
>  	unsigned long si_mask = hung_task_si_mask;
> 
>  	/*
> @@ -334,7 +339,8 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
>  			 */
>  			atomic_long_inc(&sysctl_hung_task_detect_count);
>  			this_round_count++;
> -			hung_task_info(t, timeout, this_round_count);
> +			hung_task_info(t, timeout, this_round_count,
> +				       warning_budget);
>  		}
>  	}
>   unlock:
> @@ -483,6 +489,15 @@ static const struct ctl_table hung_task_sysctls[] = {
>  		.proc_handler	= proc_dointvec_minmax,
>  		.extra1		= SYSCTL_NEG_ONE,
>  	},
> +	{
> +		.procname	= "hung_task_warnings_mode",
> +		.data		= &sysctl_hung_task_warnings_mode,
> +		.maxlen		= sizeof(int),
> +		.mode		= 0644,
> +		.proc_handler	= proc_dointvec_minmax,
> +		.extra1		= SYSCTL_ZERO,
> +		.extra2		= SYSCTL_ONE,
> +	},
>  	{
>  		.procname	= "hung_task_detect_count",
>  		.maxlen		= sizeof(unsigned long),
> ---
> 
> Per-scan mode avoids the permanent blind spot, while still letting users
> pick whichever behavior fits their setup.
> 
> Keeps the change local, too. If that turns out not to be enough, sure,
> revisit dedup then. That's the direction I'd support :D
> 
> [1] https://lore.kernel.org/lkml/akZlVgbImH6Jqy55@pathway.suse.cz/
> 
> Thanks, Lance

Hi Lance,

Thank you for taking the time to review the patch and share your thoughts.

While I understand your reservations regarding the complexity, I must
respectfully point out that implementation [1] (v6), coupled with the minor
adjustment below, already directly addresses both of the precise issues
Petr outlined, and does so without requiring a new sysctl.

[1]: https://lore.kernel.org/lkml/20260719161305.428947-1-atomlin@atomlin.com/

To Petr's first point, log bloat from multiple tasks waiting on a single
lock, your proposed hung_task_warnings_mode does not mitigate the redundant
stack traces generated within a single scan. If dozens of tasks are hung on
the identical lock, the ring buffer will still be severely flooded. The
deduplication in v6 is specifically designed to solve this fundamental
issue.

To Petr's second point, the permanent blind spot, v6 already implements a
native solution that eliminates the need for an additional mode switch. As
noted in the original commit message, the warning budget is automatically
restored once the hang resolves (i.e., when no active hung tasks are
detected in a round). This ensures system administrators are never left
permanently blind.

The following minor adjustment to implementation [1] simply relocates the
panic evaluation back to the main loop, ensuring it operates correctly
prior to the automated budget restoration logic:


diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 6bb2b26693a1..2a11d5ebcdf1 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -252,11 +252,6 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 {
 	trace_sched_process_hang(t);
 
-	if (sysctl_hung_task_panic && this_round_count >= sysctl_hung_task_panic) {
-		console_verbose();
-		hung_task_call_panic = true;
-	}
-
 	/*
 	 * The given task did not get scheduled for more than
 	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
@@ -371,6 +366,12 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 			atomic_long_inc(&sysctl_hung_task_detect_count);
 			this_round_count++;
 
+			if (sysctl_hung_task_panic &&
+			    this_round_count >= sysctl_hung_task_panic) {
+				console_verbose();
+				hung_task_call_panic = true;
+			}
+
 #ifdef CONFIG_DETECT_HUNG_TASK_BLOCKER
 			if (!hung_task_has_active) {
 				warnings_reset_value = sysctl_hung_task_warnings;


By automatically restoring the budget upon resolution, we achieve the
desired persistent visibility without burdening users to configure a new
sysctl, while crucially preventing log saturation during a massive lock
contention event.

I believe this approach provides the comprehensive fix to both of Petr's
concerns.


Kind regards,
-- 
Aaron Tomlin

Re: [PATCH v6] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[Petr Mladek]

On Tue 2026-07-21 13:08:32, Lance Yang wrote:
> 
> On Sun, Jul 19, 2026 at 12:13:05PM -0400, Aaron Tomlin wrote:
> >Currently, during severe lock contention, multiple tasks can hang while
> >waiting on the exact same resource. The khungtaskd kthread
> >indiscriminately reports every single instance with a stack trace.
> >This can roll the kernel ring buffer and prematurely exhaust the
> >kernel.hung_task_warnings budget. Consequently, the kernel is left
> >entirely blind to subsequent, unrelated deadlocks.
> >
> >To preserve the warning budget and ring buffer without sacrificing
> >observability, this patch introduces a two-tier deduplication mechanism:
> >
> >    1. Introduces a hung_task_reported in task_struct. If a task remains
> >       hung across multiple check intervals, khungtaskd suppresses
> >       redundant stack traces for that specific task until it makes
> >       progress (verified via context switch counters). Furthermore, it
> >       is packed into an existing compiler alignment hole, consuming
> >       zero additional memory
> >
> >    2. For tasks blocked on a lock, we leverage the
> >       CONFIG_DETECT_HUNG_TASK_BLOCKER infrastructure. By tracking the
> >       exact memory address of the blocker lock in a persistent array of
> >       size 32, we deduplicate warning reports for tasks waiting on the
> >       same resource across check intervals. If the blocker is already
> >       tracked, the warning is suppressed and the warning budget is
> >       preserved
> >
> >    3. For duplicate tasks, we still print the single-line "INFO: task
> >       ..." message and trigger tracepoint trace_sched_process_hang().
> >       It merely skips calling sched_show_task() and
> >       debug_show_blocker(), printing a concise suppression notice
> >       instead
> >
> >    4. Once the hang resolves (i.e., no hung tasks are detected in a
> >       check round), the budget is restored to the value captured at the
> >       start of the hang, and the blocker array is cleared, accompanied
> >       by a recovery message in the ring buffer
> >
> >Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
> >---
> 
> Sorry for the late reply.

Don't worry. IMHO, a reply within one week is perfectly
fine. Sometimes even this is not possible.

> Still, v6 looks far too complicated ... and touches too much core code
> for the problem you're trying to address ...

I agree that it looks too complicated. I see several problems:

  1. Many changes are added in a single patch. I suggest to split
     it next time into more patches and add the features from
     the most important one, ...

  2. The code seems to be a bit over-engineered. There are too many
     variables hung_task_blockers_count, hung_task_has_active,
     warnings_decremented, skip_show_task, scan_completed,
     did_report, seen_blockers_mask, ... I believe that it might
     be done an easier way.

  3. Too much spaghetti code is added to the already large
     check_hung_uninterruptible_tasks() function. It might look
     better when the detection of the duplicated task is handled
     in a separate function, ...


> Looked at this again ... Petr's earlier breakdown[1] makes sense. Both
> problems are real:
> 
> "
> I would split this into two problems:
> 
> 1. A single lock contention might trigger hung_report for many tasks
>    waiting for the same lock. It bloats the kernel log and messages
>    might even get lost.
> 
> 2. The number of printed backtraces can be reduced by a global limit.
>    But the limit silences the hung task detector and system
>    administrators are blind once the limit is reached.
> "
> 
> IMO, that's the tradeoff: keep log noise down without leaving users
> blind for good once budget runs out ...
> 
> Still not sold on dedup, though. Sure, duplicate reports happen, but how
> often, really? Doesn't seem common enough to need this much code :)
> 
> If we really want to handle this in kernel, how about a simple mode
> switch for hung_task_warnings?
> 
>   0 = keep the current global budget
>   1 = allow up to hung_task_warnings reports in each scan

Honestly, I do not think that this is the right way to go. The more I think
about it, the more I believe that the global budget is simply wrong.

IMHO, we should do the following:

1. Reset the global budget when the situation is resolved and
   there is no hung task any longer.

2. The global budget should limit only printing all the details,
   especially backtraces. We should always print at least
   the task name, ...

3. We could prevent printing the same process again and again
   by adding "hung_task_reported" into struct task_struct.

   But then we should print some summary about how many hung tasks
   were found in the last round at least. So that we know that
   the problem persists.

   This should still be rather easy. But I am not sure if it is
   worth it.

4. We could add the filtering of the duplicated backtraces. But
   it has to be handled by some helper function.

   It looks like to most complicated part. I would personally skip
   it for now. We could always add it later when the above proposed
   changes are not enough to tune the kernel.hung_task_warnings
   value in the real life.


I tried to implement the 1st and 2nd change using gemini just to
see how it would look. And it looks good to me:

From 4843170a1317f2fd3bebc1efc63c387840a4ed0d Mon Sep 17 00:00:00 2001
From: Petr Mladek <pmladek@suse.com>
Date: Tue, 21 Jul 2026 14:09:33 +0200
Subject: [PATCH 1/2] hung_task: Reset warning budget when problem gets
 resolved

sysctl_hung_task_warnings counts how many hung tasks are reported.
The watchdog does not report anything once the limit is reached.
Currently, this budget is decremented permanently, meaning the kernel is
left blind to subsequent hung tasks even after the original issue resolves.

Keep the global sysctl_hung_task_warnings intact, and instead decrement
a copy (hung_task_warnings_printed) when warnings are printed. Reset
the copy back to the configured sysctl_hung_task_warnings limit once the
problem on the system gets resolved and check_hung_uninterruptible_tasks()
detects no hung tasks in a check interval.

Also keep the copy updated when the global sysctl_hung_task_warnings
value is updated via sysctl, and update documentation to reflect
the new behavior.

Assisted-by: gemini-3.5-flash
Signed-off-by: Petr Mladek <pmladek@suse.com>
---
 Documentation/admin-guide/sysctl/kernel.rst |  6 ++--
 kernel/hung_task.c                          | 34 ++++++++++++++++-----
 2 files changed, 31 insertions(+), 9 deletions(-)

diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index c6994e55d141..eaa1d837c739 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -459,8 +459,10 @@ hung_task_warnings
 ==================
 
 The maximum number of warnings to report. During a check interval
-if a hung task is detected, this value is decreased by 1.
-When this value reaches 0, no more warnings will be reported.
+if a hung task is detected, the internal warning budget is decreased by 1.
+When this budget reaches 0, no more warnings will be reported.
+The warning budget is reset back to the configured limit when the problem on the
+system gets resolved and no hung task is found.
 This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.
 
 -1: report an infinite number of warnings.
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 6fcc94ce4ca9..4e1fb0db79d1 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -59,6 +59,8 @@ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs;
 
 static int __read_mostly sysctl_hung_task_warnings = 10;
 
+static int hung_task_warnings_printed = 10;
+
 static int __read_mostly did_panic;
 static bool hung_task_call_panic;
 
@@ -247,9 +249,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
 	 * accordingly
 	 */
-	if (sysctl_hung_task_warnings || hung_task_call_panic) {
-		if (sysctl_hung_task_warnings > 0)
-			sysctl_hung_task_warnings--;
+	if (hung_task_warnings_printed || hung_task_call_panic) {
+		if (hung_task_warnings_printed > 0)
+			hung_task_warnings_printed--;
 		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
 		       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
 		       (jiffies - t->last_switch_time) / HZ);
@@ -264,7 +266,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		sched_show_task(t);
 		debug_show_blocker(t, timeout);
 
-		if (!sysctl_hung_task_warnings)
+		if (!hung_task_warnings_printed)
 			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
 	}
 
@@ -304,7 +306,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	unsigned long last_break = jiffies;
 	struct task_struct *g, *t;
 	unsigned long this_round_count;
-	int need_warning = sysctl_hung_task_warnings;
+	int need_warning = hung_task_warnings_printed;
 	unsigned long si_mask = hung_task_si_mask;
 
 	/*
@@ -340,8 +342,10 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
  unlock:
 	rcu_read_unlock();
 
-	if (!this_round_count)
+	if (!this_round_count) {
+		hung_task_warnings_printed = sysctl_hung_task_warnings;
 		return;
+	}
 
 	if (need_warning || hung_task_call_panic) {
 		si_mask |= SYS_INFO_LOCKS;
@@ -425,6 +429,22 @@ static int proc_dohung_task_timeout_secs(const struct ctl_table *table, int writ
 	return ret;
 }
 
+static int proc_dohung_task_warnings(const struct ctl_table *table, int write,
+				     void *buffer,
+				     size_t *lenp, loff_t *ppos)
+{
+	int ret;
+
+	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
+
+	if (ret || !write)
+		return ret;
+
+	hung_task_warnings_printed = sysctl_hung_task_warnings;
+
+	return 0;
+}
+
 /*
  * This is needed for proc_doulongvec_minmax of sysctl_hung_task_timeout_secs
  * and hung_task_check_interval_secs
@@ -480,7 +500,7 @@ static const struct ctl_table hung_task_sysctls[] = {
 		.data		= &sysctl_hung_task_warnings,
 		.maxlen		= sizeof(int),
 		.mode		= 0644,
-		.proc_handler	= proc_dointvec_minmax,
+		.proc_handler	= proc_dohung_task_warnings,
 		.extra1		= SYSCTL_NEG_ONE,
 	},
 	{
-- 
2.55.0



From 50f69511d706484d9378c019f8cbaa3232555fa6 Mon Sep 17 00:00:00 2001
From: Petr Mladek <pmladek@suse.com>
Date: Tue, 21 Jul 2026 14:49:16 +0200
Subject: [PATCH 2/2] hung_task: Always print basic hung task info header

Currently, hung_task_info() completely suppresses any warning messages
once the warning budget is reached. This leaves the system completely
blind to any future hung tasks, even though we only wanted to suppress
heavy process stack dumps.

Modify hung_task_info() to always print the basic single-line error message
about a hung task, regardless of whether the warning budget has been
exhausted. Restrict only the verbose details (such as stack dumps,
taint/release status, and blockers) to when the warning budget is intact
or panic is triggered.

Additionally, update the notice printed upon warning exhaustion to clarify
that future reports will only omit process details rather than being completely
suppressed.

Assisted-by: gemini-3.5-flash
Signed-off-by: Petr Mladek <pmladek@suse.com>
---
 kernel/hung_task.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 4e1fb0db79d1..6ebb3a87ac65 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -244,17 +244,19 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		hung_task_call_panic = true;
 	}
 
+	/* Always print the blocked message */
+	pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
+	       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
+	       (jiffies - t->last_switch_time) / HZ);
+
 	/*
 	 * The given task did not get scheduled for more than
 	 * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
-	 * accordingly
+	 * accordingly with full details if the budget is not exhausted.
 	 */
 	if (hung_task_warnings_printed || hung_task_call_panic) {
 		if (hung_task_warnings_printed > 0)
 			hung_task_warnings_printed--;
-		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
-		       t->comm, t->pid, t->in_iowait ? " in I/O wait" : "",
-		       (jiffies - t->last_switch_time) / HZ);
 		pr_err("      %s %s %.*s\n",
 			print_tainted(), init_utsname()->release,
 			(int)strcspn(init_utsname()->version, " "),
@@ -267,7 +269,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 		debug_show_blocker(t, timeout);
 
 		if (!hung_task_warnings_printed)
-			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
+			pr_info("Future hung task reports won't print details about each process, see sysctl kernel.hung_task_warnings\n");
 	}
 
 	touch_nmi_watchdog();
-- 
2.55.0

Re: [PATCH v6] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[Lance Yang]

On Tue, Jul 21, 2026 at 03:28:48PM +0200, Petr Mladek wrote:
>On Tue 2026-07-21 13:08:32, Lance Yang wrote:
>> 
>> On Sun, Jul 19, 2026 at 12:13:05PM -0400, Aaron Tomlin wrote:
[...]
>> 
>> Sorry for the late reply.
>
>Don't worry. IMHO, a reply within one week is perfectly
>fine. Sometimes even this is not possible.

Cheers!

>> Still, v6 looks far too complicated ... and touches too much core code
>> for the problem you're trying to address ...
>
>I agree that it looks too complicated. I see several problems:
>
>  1. Many changes are added in a single patch. I suggest to split
>     it next time into more patches and add the features from
>     the most important one, ...
>
>  2. The code seems to be a bit over-engineered. There are too many
>     variables hung_task_blockers_count, hung_task_has_active,
>     warnings_decremented, skip_show_task, scan_completed,
>     did_report, seen_blockers_mask, ... I believe that it might
>     be done an easier way.
>
>  3. Too much spaghetti code is added to the already large
>     check_hung_uninterruptible_tasks() function. It might look
>     better when the detection of the duplicated task is handled
>     in a separate function, ...

v6 kinda snowballed :) Way too much state for what it buys us :)

>
>> Looked at this again ... Petr's earlier breakdown[1] makes sense. Both
>> problems are real:
>> 
>> "
>> I would split this into two problems:
>> 
>> 1. A single lock contention might trigger hung_report for many tasks
>>    waiting for the same lock. It bloats the kernel log and messages
>>    might even get lost.
>> 
>> 2. The number of printed backtraces can be reduced by a global limit.
>>    But the limit silences the hung task detector and system
>>    administrators are blind once the limit is reached.
>> "
>> 
>> IMO, that's the tradeoff: keep log noise down without leaving users
>> blind for good once budget runs out ...
>> 
>> Still not sold on dedup, though. Sure, duplicate reports happen, but how
>> often, really? Doesn't seem common enough to need this much code :)
>> 
>> If we really want to handle this in kernel, how about a simple mode
>> switch for hung_task_warnings?
>> 
>>   0 = keep the current global budget
>>   1 = allow up to hung_task_warnings reports in each scan
>
>Honestly, I do not think that this is the right way to go. The more I think
>about it, the more I believe that the global budget is simply wrong.
>
>IMHO, we should do the following:
>
>1. Reset the global budget when the situation is resolved and
>   there is no hung task any longer.
>
>2. The global budget should limit only printing all the details,
>   especially backtraces. We should always print at least
>   the task name, ...

Yeah, sounds better than another mode. Slight semantic change, but looks
fine to me.

>3. We could prevent printing the same process again and again
>   by adding "hung_task_reported" into struct task_struct.
>
>   But then we should print some summary about how many hung tasks
>   were found in the last round at least. So that we know that
>   the problem persists.
>
>   This should still be rather easy. But I am not sure if it is
>   worth it.

With 1 and 2 in place, don't think this is needed anymore ;)

>4. We could add the filtering of the duplicated backtraces. But
>   it has to be handled by some helper function.
>
>   It looks like to most complicated part. I would personally skip
>   it for now. We could always add it later when the above proposed
>   changes are not enough to tune the kernel.hung_task_warnings
>   value in the real life.

And blocker dedup gets even messier. Haven't seen a simple way to do
that. Happy to skip it :D

>
>
>I tried to implement the 1st and 2nd change using gemini just to
>see how it would look. And it looks good to me:

I really like this approach :)

Hope v7 sticks to these two changes; think that's all we need :)

[...]

Cheers, Lance

Re: [PATCH v6] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[Aaron Tomlin]

On Tue, Jul 21, 2026 at 03:28:48PM +0200, Petr Mladek wrote:
> > Still, v6 looks far too complicated ... and touches too much core code
> > for the problem you're trying to address ...
> 
> I agree that it looks too complicated. I see several problems:
> 
>   1. Many changes are added in a single patch. I suggest to split
>      it next time into more patches and add the features from
>      the most important one, ...
> 
>   2. The code seems to be a bit over-engineered. There are too many
>      variables hung_task_blockers_count, hung_task_has_active,
>      warnings_decremented, skip_show_task, scan_completed,
>      did_report, seen_blockers_mask, ... I believe that it might
>      be done an easier way.
> 
>   3. Too much spaghetti code is added to the already large
>      check_hung_uninterruptible_tasks() function. It might look
>      better when the detection of the duplicated task is handled
>      in a separate function, ...

Hi Petr,

Thank you for the detailed review and for taking the time to draft these
patches. I completely agree with your assessment; in my attempt to cover
every edge case of lock contention, the v6 implementation became
too entangled within check_hung_uninterruptible_tasks().

Your approach in these two patches is much more elegant. Decoupling the
active warning budget from the configured sysctl limit perfectly solves the
permanent blind spot, and ensuring the basic one-line report always prints
provides sufficient visibility without the risk of massive stack-dump log
bloat.

I will happily concede.

> 3. We could prevent printing the same process again and again
>    by adding "hung_task_reported" into struct task_struct.
> 
>    But then we should print some summary about how many hung tasks
>    were found in the last round at least. So that we know that
>    the problem persists.
> 
>    This should still be rather easy. But I am not sure if it is
>    worth it.
> 
> 4. We could add the filtering of the duplicated backtraces. But
>    it has to be handled by some helper function.
> 
>    It looks like to most complicated part. I would personally skip
>    it for now. We could always add it later when the above proposed
>    changes are not enough to tune the kernel.hung_task_warnings
>    value in the real life.

I do think this is worth doing to prevent the single-line spam if a task
remains hung for hours, no?


Kind regards,
-- 
Aaron Tomlin