[PATCH v2] hung_task: Add per-round stack trace deduplication

[PATCH v2] hung_task: Add per-round stack trace deduplication

[Aaron Tomlin]

Currently, when multiple tasks hang in the exact same location (e.g.,
such as severe contention for a mutex), khungtaskd indiscriminately
reports every single instance. This wastes ring buffer space with
identical stack traces up to the defined warning limit (i.e.,
kernel.hung_task_warnings), obscuring the root cause without providing
any additional diagnostic value.

Introduce a lightweight, hash-based stack trace deduplicator for
khungtaskd to ensure only unique stack traces are reported during
a single detection interval.

Technical details of the implementation:
 - Uses a 12-bit hash table (4096 slots), consuming just 16 KB of
   static memory to prevent cache thrashing during massive hangs.

 - Operates purely serially within the single khungtaskd thread,
   requiring zero atomic operations or concurrent locking overhead.

 - Flushes the lossy cache via memset() at the beginning of each
   detection round. This ensures the immediate "thundering herd" of
   duplicates is suppressed, but guarantees the system will not
   permanently suppress identical hangs that occur in future rounds.

 - Introduces a new sysctl, kernel.hung_task_dedup, which defaults to 1
   (enabled). The sysctl is locally cached at the outset of each
   interval to prevent tearing caused by concurrent userspace toggling.

Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
--
Changes since v1:

 - Preserve "INFO:" headers for all hung tasks; suppress only the stack
   dumps for duplicates (Masami Hiramatsu)

 - Print a clear notification when a trace is explicitly suppressed

 - Add #ifdef CONFIG_STACKTRACE guards to prevent Kconfig build errors

 - Optimise overhead by unwinding the stack only if a warning is
   actually going to be printed

 - Linked to v1: https://lore.kernel.org/lkml/20260617184841.1447955-1-atomlin@atomlin.com/
---
 kernel/hung_task.c | 108 ++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 103 insertions(+), 5 deletions(-)

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 6fcc94ce4ca9..ccd42f0a27ab 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -25,6 +25,9 @@
 #include <linux/hung_task.h>
 #include <linux/rwsem.h>
 #include <linux/sys_info.h>
+#include <linux/stacktrace.h>
+#include <linux/jhash.h>
+#include <linux/hash.h>
 
 #include <trace/events/sched.h>
 
@@ -59,6 +62,23 @@ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs;
 
 static int __read_mostly sysctl_hung_task_warnings = 10;
 
+#ifdef CONFIG_STACKTRACE
+/*
+ * Sizing the deduplicator hash table.
+ * 12 bits provides 4096 slots, costing just 16 KB of static memory.
+ */
+#define HUNG_TASK_HT_BITS 12
+#define HUNG_TASK_HT_SIZE (1UL << HUNG_TASK_HT_BITS)
+
+static u32 hung_task_hash_table[HUNG_TASK_HT_SIZE];
+
+/*
+ * Enable or disable stack trace deduplication.
+ * Defaults to 1 (enabled).
+ */
+static int __read_mostly sysctl_hung_task_dedup = 1;
+#endif
+
 static int __read_mostly did_panic;
 static bool hung_task_call_panic;
 
@@ -223,17 +243,70 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti
 }
 #endif
 
+#ifdef CONFIG_STACKTRACE
+/**
+ * hung_task_stack_is_unique - Check if a stack trace has been seen this round
+ * @t: Pointer to the task structure
+ *
+ * Captures the stack trace of the task, hashes it, and checks our lossy
+ * cache. Since this is only called serially from the khungtaskd thread,
+ * no concurrent locks or atomics are required. Returns true if the stack
+ * is unique (or we collided), false if it is a known duplicate.
+ */
+static bool hung_task_stack_is_unique(struct task_struct *t)
+{
+	unsigned long entries[64];
+	unsigned int nr_entries;
+	u32 hash, idx;
+
+	nr_entries = stack_trace_save_tsk(t, entries, ARRAY_SIZE(entries), 0);
+	hash = jhash2((u32 *)entries, nr_entries * (sizeof(unsigned long) /
+		      sizeof(u32)), JHASH_INITVAL);
+
+	if (unlikely(!hash))
+		hash = 1;
+
+	idx = hash_32(hash, HUNG_TASK_HT_BITS);
+
+	if (hung_task_hash_table[idx] == hash)
+		return false;
+
+	hung_task_hash_table[idx] = hash;
+
+	return true;
+}
+
+static inline void hung_task_dedup_flush(void)
+{
+	memset(hung_task_hash_table, 0, sizeof(hung_task_hash_table));
+}
+#else
+static bool hung_task_stack_is_unique(struct task_struct *t)
+{
+	return true;
+}
+
+static inline void hung_task_dedup_flush(void)
+{
+}
+#endif /* CONFIG_STACKTRACE */
+
 /**
  * hung_task_info - Print diagnostic details for a hung task
  * @t: Pointer to the detected hung task.
  * @timeout: Timeout threshold for detecting hung tasks
  * @this_round_count: Count of hung tasks detected in the current iteration
+ * @dedup_enabled: Snapshot of sysctl_hung_task_dedup for the current interval.
  *
  * Print structured information about the specified hung task, if warnings
- * are enabled or if the panic batch threshold is exceeded.
+ * are enabled or if the panic batch threshold is exceeded. If @dedup_enabled
+ * is true, identical stack traces within the same detection round are
+ * intercepted and suppressed to save ring buffer space.
  */
 static void hung_task_info(struct task_struct *t, unsigned long timeout,
-			   unsigned long this_round_count)
+			   unsigned long this_round_count,
+			   bool dedup_enabled)
+
 {
 	trace_sched_process_hang(t);
 
@@ -261,8 +334,14 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 			pr_err("      Blocked by coredump.\n");
 		pr_err("\"echo 0 > /proc/sys/kernel/hung_task_timeout_secs\""
 			" disables this message.\n");
-		sched_show_task(t);
-		debug_show_blocker(t, timeout);
+
+		/* Intercept and drop duplicate stack traces */
+		if (!dedup_enabled || hung_task_stack_is_unique(t)) {
+			sched_show_task(t);
+			debug_show_blocker(t, timeout);
+		} else {
+			pr_err("      Identical stack trace suppressed. See sysctl kernel.hung_task_dedup\n");
+		}
 
 		if (!sysctl_hung_task_warnings)
 			pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n");
@@ -306,6 +385,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	unsigned long this_round_count;
 	int need_warning = sysctl_hung_task_warnings;
 	unsigned long si_mask = hung_task_si_mask;
+	bool dedup_enabled = IS_ENABLED(CONFIG_STACKTRACE) ? READ_ONCE(sysctl_hung_task_dedup) : 0;
 
 	/*
 	 * If the system crashed already then all bets are off,
@@ -314,6 +394,13 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 	if (test_taint(TAINT_DIE) || did_panic)
 		return;
 
+	/*
+	 * Reset the de-duplicator hash table for this new detection round.
+	 * This prevents stale hashes from permanently suppressing.
+	 */
+	if (dedup_enabled)
+		hung_task_dedup_flush();
+
 	this_round_count = 0;
 	rcu_read_lock();
 	for_each_process_thread(g, t) {
@@ -334,7 +421,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
 			 */
 			atomic_long_inc(&sysctl_hung_task_detect_count);
 			this_round_count++;
-			hung_task_info(t, timeout, this_round_count);
+			hung_task_info(t, timeout, this_round_count, dedup_enabled);
 		}
 	}
  unlock:
@@ -483,6 +570,17 @@ static const struct ctl_table hung_task_sysctls[] = {
 		.proc_handler	= proc_dointvec_minmax,
 		.extra1		= SYSCTL_NEG_ONE,
 	},
+#ifdef CONFIG_STACKTRACE
+	{
+		.procname	= "hung_task_dedup",
+		.data		= &sysctl_hung_task_dedup,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= SYSCTL_ZERO,
+		.extra2		= SYSCTL_ONE,
+	},
+#endif
 	{
 		.procname	= "hung_task_detect_count",
 		.maxlen		= sizeof(unsigned long),
-- 
2.51.0

Re: [PATCH v2] hung_task: Add per-round stack trace deduplication

[Lance Yang]

Hi Aaron,

On Fri, Jun 19, 2026 at 09:35:59PM -0400, Aaron Tomlin wrote:
>Currently, when multiple tasks hang in the exact same location (e.g.,
>such as severe contention for a mutex), khungtaskd indiscriminately
>reports every single instance. This wastes ring buffer space with
>identical stack traces up to the defined warning limit (i.e.,
>kernel.hung_task_warnings), obscuring the root cause without providing
>any additional diagnostic value.
>
>Introduce a lightweight, hash-based stack trace deduplicator for
>khungtaskd to ensure only unique stack traces are reported during
>a single detection interval.
>
>Technical details of the implementation:
> - Uses a 12-bit hash table (4096 slots), consuming just 16 KB of
>   static memory to prevent cache thrashing during massive hangs.
>
> - Operates purely serially within the single khungtaskd thread,
>   requiring zero atomic operations or concurrent locking overhead.
>
> - Flushes the lossy cache via memset() at the beginning of each
>   detection round. This ensures the immediate "thundering herd" of
>   duplicates is suppressed, but guarantees the system will not
>   permanently suppress identical hangs that occur in future rounds.
>
> - Introduces a new sysctl, kernel.hung_task_dedup, which defaults to 1
>   (enabled). The sysctl is locally cached at the outset of each
>   interval to prevent tearing caused by concurrent userspace toggling.
>

Thanks for working on this, but ... guess I'll be the bad guy here, not
convinced this should go in ...

When khungtaskd fires, somthing is already wrong, no? I don't see why it
should grow a new sysctl, a stack hash table, and extra filtering logic
just ot hide part of the report ...

Emm ... do you have real cases where duplicate hung-task stacks caused
serious pain?

If many tasks hang at once, usually one root cause, not a bunch of
different bugs. At least from what I've seen, any one of those stacks is
enough to start debugging ...

We already have hung_task_detect_count and trace_sched_process_hang() for
basic counting/observability. Even if hung_task_warnings is finite and
the warning budget runs out, we still don't lose detections: counter gets
bumped and tracepoint fires before printk output is gated :)

If someone wants stack grouping, I'd rather leave that to a tool than add
another policy knob to khungtaskd. Once it lands, maintainers have to
carry it forever. Not every nice-to-have feature is worth that cost, IMHO

And if someone really wants more hung-task stacks in the log, we already
have hung_task_warnings for that. Raise it, or set it to -1.

Also, looking at the v1 thread, I don't think the concerns there have
really settled yet ... If nobody replies, maybe give it a week before
sending a new version.

Thanks, Lance

Re: [PATCH v2] hung_task: Add per-round stack trace deduplication

[Aaron Tomlin]

[-- Attachment #1: Type: text/plain, Size: 6053 bytes --]

On Sat, Jun 20, 2026 at 11:37:15AM +0800, Lance Yang wrote:
> Hi Aaron,
> 
> On Fri, Jun 19, 2026 at 09:35:59PM -0400, Aaron Tomlin wrote:
> >Currently, when multiple tasks hang in the exact same location (e.g.,
> >such as severe contention for a mutex), khungtaskd indiscriminately
> >reports every single instance. This wastes ring buffer space with
> >identical stack traces up to the defined warning limit (i.e.,
> >kernel.hung_task_warnings), obscuring the root cause without providing
> >any additional diagnostic value.
> >
> >Introduce a lightweight, hash-based stack trace deduplicator for
> >khungtaskd to ensure only unique stack traces are reported during
> >a single detection interval.
> >
> >Technical details of the implementation:
> > - Uses a 12-bit hash table (4096 slots), consuming just 16 KB of
> >   static memory to prevent cache thrashing during massive hangs.
> >
> > - Operates purely serially within the single khungtaskd thread,
> >   requiring zero atomic operations or concurrent locking overhead.
> >
> > - Flushes the lossy cache via memset() at the beginning of each
> >   detection round. This ensures the immediate "thundering herd" of
> >   duplicates is suppressed, but guarantees the system will not
> >   permanently suppress identical hangs that occur in future rounds.
> >
> > - Introduces a new sysctl, kernel.hung_task_dedup, which defaults to 1
> >   (enabled). The sysctl is locally cached at the outset of each
> >   interval to prevent tearing caused by concurrent userspace toggling.
> >
> 
> Thanks for working on this, but ... guess I'll be the bad guy here, not
> convinced this should go in ...
> 
> When khungtaskd fires, somthing is already wrong, no? I don't see why it
> should grow a new sysctl, a stack hash table, and extra filtering logic
> just ot hide part of the report ...
> 
> Emm ... do you have real cases where duplicate hung-task stacks caused
> serious pain?
> 
> If many tasks hang at once, usually one root cause, not a bunch of
> different bugs. At least from what I've seen, any one of those stacks is
> enough to start debugging ...
> 
> We already have hung_task_detect_count and trace_sched_process_hang() for
> basic counting/observability. Even if hung_task_warnings is finite and
> the warning budget runs out, we still don't lose detections: counter gets
> bumped and tracepoint fires before printk output is gated :)
> 
> If someone wants stack grouping, I'd rather leave that to a tool than add
> another policy knob to khungtaskd. Once it lands, maintainers have to
> carry it forever. Not every nice-to-have feature is worth that cost, IMHO
> 
> And if someone really wants more hung-task stacks in the log, we already
> have hung_task_warnings for that. Raise it, or set it to -1.
> 
> Also, looking at the v1 thread, I don't think the concerns there have
> really settled yet ... If nobody replies, maybe give it a week before
> sending a new version.

Hi Lance,

Thank you for taking the time to review the patch and for your candour.

You raise an entirely fair point regarding maintainability; every new
control knob indeed carries a permanent cost for the maintainers, and I
respect your caution.

To answer your question regarding real world pain: the primary issue is not
merely visual clutter, but the premature exhaustion of the warning budget
and the preservation of the kernel ring buffer during cascading failures.

In our production environments, we typically leave
kernel.hung_task_warnings at its default value of 10. If a severe lock
contention occurs, a single bottleneck can easily cause 10 tasks to hang
simultaneously with the exact same stack trace. Under the current logic,
those 10 identical traces will completely exhaust the warning budget.
Consequently, the kernel is left entirely blind to any subsequent or
completely unrelated deadlocks that might be occurring concurrently, as all
further reports are silenced.

Furthermore, dumping a full stack trace for every duplicate rapidly injects
several of lines of identical noise into dmesg. We have found that this
sudden burst frequently rolls the circular ring buffer.

Userspace tooling is unfortunately unable to group or analyse logs that
have already been evicted before the tool could read them, nor can it
recover traces the kernel silently dropped due to an exhausted budget.

The deduplicator acts as a telemetry filter, ensuring that the limited
warning budget is spent strictly on unique traces rather than redundant
noise, thereby preserving the history of the crash and ensuring secondary
failures are not obscured.

I wanted to clarify the exact operational context and the limitation of
relying on userspace. Please let me know if this operational context alters
your perspective at all.

Please note Sashiko [1] flagged a few issues which will be addressed by the
following changes:

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index ccd42f0a27ab..51f6540b70b3 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -77,6 +77,8 @@ static u32 hung_task_hash_table[HUNG_TASK_HT_SIZE];
  * Defaults to 1 (enabled).
  */
 static int __read_mostly sysctl_hung_task_dedup = 1;
+#else
+#define sysctl_hung_task_dedup 0
 #endif
 
 static int __read_mostly did_panic;
@@ -258,8 +260,18 @@ static bool hung_task_stack_is_unique(struct task_struct *t)
 	unsigned long entries[64];
 	unsigned int nr_entries;
 	u32 hash, idx;
+	void *stack;
+
+	/* Pin the stack safely before unwinding */
+	stack = try_get_task_stack(t);
+	if (!stack)
+		/* Task is exiting; pretend it is unique it */
+		return true;
 
 	nr_entries = stack_trace_save_tsk(t, entries, ARRAY_SIZE(entries), 0);
+
+	put_task_stack(t);
+
 	hash = jhash2((u32 *)entries, nr_entries * (sizeof(unsigned long) /
 		      sizeof(u32)), JHASH_INITVAL);
 


[1]: https://sashiko.dev/#/patchset/20260620013559.1537893-1-atomlin%40atomlin.com

Kind regards,
-- 
Aaron Tomlin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2] hung_task: Add per-round stack trace deduplication

[Lance Yang]

On Sat, Jun 20, 2026 at 01:54:56PM -0400, Aaron Tomlin wrote:
[...]
>On Sat, Jun 20, 2026 at 11:37:15AM +0800, Lance Yang wrote:
>> Hi Aaron,
>>
>> On Fri, Jun 19, 2026 at 09:35:59PM -0400, Aaron Tomlin wrote:
>> >Currently, when multiple tasks hang in the exact same location (e.g.,
>> >such as severe contention for a mutex), khungtaskd indiscriminately
>> >reports every single instance. This wastes ring buffer space with
>> >identical stack traces up to the defined warning limit (i.e.,
>> >kernel.hung_task_warnings), obscuring the root cause without providing
>> >any additional diagnostic value.
>> >
>> >Introduce a lightweight, hash-based stack trace deduplicator for
>> >khungtaskd to ensure only unique stack traces are reported during
>> >a single detection interval.
>> >
>> >Technical details of the implementation:
>> > - Uses a 12-bit hash table (4096 slots), consuming just 16 KB of
>> >   static memory to prevent cache thrashing during massive hangs.
>> >
>> > - Operates purely serially within the single khungtaskd thread,
>> >   requiring zero atomic operations or concurrent locking overhead.
>> >
>> > - Flushes the lossy cache via memset() at the beginning of each
>> >   detection round. This ensures the immediate "thundering herd" of
>> >   duplicates is suppressed, but guarantees the system will not
>> >   permanently suppress identical hangs that occur in future rounds.
>> >
>> > - Introduces a new sysctl, kernel.hung_task_dedup, which defaults to 1
>> >   (enabled). The sysctl is locally cached at the outset of each
>> >   interval to prevent tearing caused by concurrent userspace toggling.
>> >
>>
>> Thanks for working on this, but ... guess I'll be the bad guy here, not
>> convinced this should go in ...
>>
>> When khungtaskd fires, somthing is already wrong, no? I don't see why it
>> should grow a new sysctl, a stack hash table, and extra filtering logic
>> just ot hide part of the report ...
>>
>> Emm ... do you have real cases where duplicate hung-task stacks caused
>> serious pain?
>>
>> If many tasks hang at once, usually one root cause, not a bunch of
>> different bugs. At least from what I've seen, any one of those stacks is
>> enough to start debugging ...
>>
>> We already have hung_task_detect_count and trace_sched_process_hang() for
>> basic counting/observability. Even if hung_task_warnings is finite and
>> the warning budget runs out, we still don't lose detections: counter gets
>> bumped and tracepoint fires before printk output is gated :)
>>
>> If someone wants stack grouping, I'd rather leave that to a tool than add
>> another policy knob to khungtaskd. Once it lands, maintainers have to
>> carry it forever. Not every nice-to-have feature is worth that cost, IMHO
>>
>> And if someone really wants more hung-task stacks in the log, we already
>> have hung_task_warnings for that. Raise it, or set it to -1.
>>
>> Also, looking at the v1 thread, I don't think the concerns there have
>> really settled yet ... If nobody replies, maybe give it a week before
>> sending a new version.
>
>Hi Lance,
>
>Thank you for taking the time to review the patch and for your candour.

I think your reply still misses most of my concerns ...

>You raise an entirely fair point regarding maintainability; every new
>control knob indeed carries a permanent cost for the maintainers, and I
>respect your caution.

Yeah, that matters, IMHO.

>To answer your question regarding real world pain: the primary issue is not
>merely visual clutter, but the premature exhaustion of the warning budget
>and the preservation of the kernel ring buffer during cascading failures.

Right, but that still sounds like a very specific case.

When khungtaskd fires, something is already wrong, no?

Even with one identical stack, per-round dedup only helps inside one
scan. The same stack can still come back in later rounds and burn through
hung_task_warnings anyway.

And under heavy contention, I would not expect only one stack anyway.
Different tasks can hang behind different locks or different callers, and
those stacks can still burn through the warning budget.

>In our production environments, we typically leave
>kernel.hung_task_warnings at its default value of 10. If a severe lock
>contention occurs, a single bottleneck can easily cause 10 tasks to hang
>simultaneously with the exact same stack trace. Under the current logic,

Not sure I buy this premise :)

Same bottleneck does not necessarily mean exactly the same stack.
Different callers can block on the same lock, and exact-stack dedup won't
help there.

At least from cases I've looked at, I can't really recall seeing this
exact pattern often enough to justify a new khungtaskd knob.

>those 10 identical traces will completely exhaust the warning budget.
>Consequently, the kernel is left entirely blind to any subsequent or
>completely unrelated deadlocks that might be occurring concurrently, as all
>further reports are silenced.

I don't think "entirely blind" is accurate.

hung_task_warnings *only* gates printk. We still bump
hung_task_detect_count and hit trace_sched_process_hang() before that
gate.

>Furthermore, dumping a full stack trace for every duplicate rapidly injects
>several of lines of identical noise into dmesg. We have found that this
>sudden burst frequently rolls the circular ring buffer.
>
>Userspace tooling is unfortunately unable to group or analyse logs that
>have already been evicted before the tool could read them, nor can it
>recover traces the kernel silently dropped due to an exhausted budget.
>
>The deduplicator acts as a telemetry filter, ensuring that the limited
>warning budget is spent strictly on unique traces rather than redundant
>noise, thereby preserving the history of the crash and ensuring secondary
>failures are not obscured.
>
>I wanted to clarify the exact operational context and the limitation of
>relying on userspace. Please let me know if this operational context alters
>your perspective at all.
[...]

Aaron, you've done good work in khungtaskd, and some of it is upstream
already. I do appreciate that!

But this one feels different. Useful locally, maybe, but not something
the kernel should carry forever.

Anyway, I'll stop here. Still a nack from my side.

Thanks, Lance