Re: [PATCH v4] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[Petr Mladek <pmladek@suse.com>]

On Sun 2026-06-28 16:56:39, Aaron Tomlin wrote:
> On Sun, Jun 28, 2026 at 12:47:50PM +0800, Lance Yang wrote:
> To step back and address your question directly regarding the real-world
> problem this patch aims to solve:

Thanks a lot for slowing down. It allows people to think more about
the problem and get feedback from more poeple. And it reduces the risk
of burn out of maintainers and reviewers.

> In large-scale, multi-tenant, production environments, lock contention is a
> frequent reality. When a core resource (e.g., a heavily contended rwsem or
> mutex) blocks, it does not just hang one task; it causes a cascading
> failure that halts hundreds of tasks simultaneously.
> 
> When khungtaskd runs its scan during such an outage, it often reports
> identical stack traces into the kernel ring buffer, which is not entirely
> useful.
> 
> The global sysctl_hung_task_warnings budget is instantly exhausted by a
> single lock storm. Consequently, the kernel is left entirely blind to
> subsequent, completely unrelated deadlocks occurring elsewhere in the
> system hours later.
> 
> The changes introduced to date, moving away from the heuristic wchan
> approach to a more deterministic t->blocker tracking as per Petr's
> feedback, were an attempt to solve this without introducing complex
> heuristics or dangerous blind spots.

I would split this into two problems:

1. A single lock contention might trigger hung_report for many tasks
   waiting for the same lock. It bloats the kernel log and messages
   might even get lost.

2. The number of printed backtraces can be reduced by a global limit.
   But the limit silences the hung task detector and system
   administrators are blind once the limit is reached.

IMHO, the global limit "sysctl_hung_task_warnings" has been introduced
because of the 1st problem but it caused the 2nd problem.

My proposal:
------------

a) We could Change the semantic of "sysctl_hung_task_warnings". It could newly
   limit the number of printed backtraces in a single hung-system
   situations. I mean to reset it when the check_hung_uninterruptible_tasks()
   does not detect any blocked tasks.

   We could even print a message in this case. Something like:

	if (atomic_long_read(&sysctl_hung_task_detect_count) && !this_round_count) {
		pr_err("INFO: Tasks are not blocked by sleeping locks any longer.\n");
		atomic_long_set(&sysctl_hung_task_detect_count, 0);
	}

   This would keep it working for 1st problem and solve the 2nd problem.


b) I like the check of task->blocker when it is available. But it
   depends on CONFIG_DETECT_HUNG_TASK_BLOCKER. Also the hash array
   looks like an overkill to me.

   I would replace the hash array with a simple array[10]. It
   should be enough in practice. Also it would be much easier
   to clear it when the hung situation has gone.

   That said, I am not sure if it is worth it.


c) Also storing the info about printed backtraces into struct
   task_struct is interesting idea.

   But again, I am not sure if it is worth it.


My opinion:
-----------

I would start with a). It is trivial. It solves the regression caused
by the current global limit. And the message about that
the hung-situation has been resolved is useful. So it
looks like win-win solution.

I would do b) and/or c) only when a) is not enough in practice.

That said:
----------

IMHO, CONFIG_DETECT_HUNG_TASK_BLOCKER is a rather cheap feature.
I believe that the overhead is small especially when we are
talking about sleeping locks. It is even enabled by default.

Adding the filtering by the blocker might be more effective
in practice than the "sysctl_hung_task_warnings" global
limit.

Best Regards,
Petr