Re: [PATCH v4] hung_task: Deduplicate identical hang reports using explicit blocker tracking

[Petr Mladek <pmladek@suse.com>]

On Sat 2026-07-04 17:35:29, Aaron Tomlin wrote:
> On Thu, Jul 02, 2026 at 03:19:18PM +0200, Petr Mladek wrote:
> > On Sun 2026-06-28 16:56:39, Aaron Tomlin wrote:
> > > On Sun, Jun 28, 2026 at 12:47:50PM +0800, Lance Yang wrote:
> > > To step back and address your question directly regarding the real-world
> > > problem this patch aims to solve:
> > 
> > Thanks a lot for slowing down. It allows people to think more about
> > the problem and get feedback from more poeple. And it reduces the risk
> > of burn out of maintainers and reviewers.
> > 
> > > In large-scale, multi-tenant, production environments, lock contention is a
> > > frequent reality. When a core resource (e.g., a heavily contended rwsem or
> > > mutex) blocks, it does not just hang one task; it causes a cascading
> > > failure that halts hundreds of tasks simultaneously.
> > > 
> > > When khungtaskd runs its scan during such an outage, it often reports
> > > identical stack traces into the kernel ring buffer, which is not entirely
> > > useful.
> > > 
> > > The global sysctl_hung_task_warnings budget is instantly exhausted by a
> > > single lock storm. Consequently, the kernel is left entirely blind to
> > > subsequent, completely unrelated deadlocks occurring elsewhere in the
> > > system hours later.
> > > 
> > > The changes introduced to date, moving away from the heuristic wchan
> > > approach to a more deterministic t->blocker tracking as per Petr's
> > > feedback, were an attempt to solve this without introducing complex
> > > heuristics or dangerous blind spots.
> > 
> > I would split this into two problems:
> > 
> > 1. A single lock contention might trigger hung_report for many tasks
> >    waiting for the same lock. It bloats the kernel log and messages
> >    might even get lost.
> > 
> > 2. The number of printed backtraces can be reduced by a global limit.
> >    But the limit silences the hung task detector and system
> >    administrators are blind once the limit is reached.
> > 
> > IMHO, the global limit "sysctl_hung_task_warnings" has been introduced
> > because of the 1st problem but it caused the 2nd problem.
> > 
> > My proposal:
> > ------------
> > 
> > a) We could Change the semantic of "sysctl_hung_task_warnings". It could newly
> >    limit the number of printed backtraces in a single hung-system
> >    situations. I mean to reset it when the check_hung_uninterruptible_tasks()
> >    does not detect any blocked tasks.
> > 
> >    We could even print a message in this case. Something like:
> > 
> > 	if (atomic_long_read(&sysctl_hung_task_detect_count) && !this_round_count) {
> > 		pr_err("INFO: Tasks are not blocked by sleeping locks any longer.\n");
> > 		atomic_long_set(&sysctl_hung_task_detect_count, 0);
> > 	}
> > 
> >    This would keep it working for 1st problem and solve the 2nd problem.
> > 
> > 
> > b) I like the check of task->blocker when it is available. But it
> >    depends on CONFIG_DETECT_HUNG_TASK_BLOCKER. Also the hash array
> >    looks like an overkill to me.
> > 
> >    I would replace the hash array with a simple array[10]. It
> >    should be enough in practice. Also it would be much easier
> >    to clear it when the hung situation has gone.
> > 
> >    That said, I am not sure if it is worth it.
> > 
> > 
> > c) Also storing the info about printed backtraces into struct
> >    task_struct is interesting idea.
> > 
> >    But again, I am not sure if it is worth it.
> > 
> > 
> > My opinion:
> > -----------
> > 
> > I would start with a). It is trivial. It solves the regression caused
> > by the current global limit. And the message about that
> > the hung-situation has been resolved is useful. So it
> > looks like win-win solution.
> > 
> > I would do b) and/or c) only when a) is not enough in practice.
> > 
> > That said:
> > ----------
> > 
> > IMHO, CONFIG_DETECT_HUNG_TASK_BLOCKER is a rather cheap feature.
> > I believe that the overhead is small especially when we are
> > talking about sleeping locks. It is even enabled by default.
> > 
> > Adding the filtering by the blocker might be more effective
> > in practice than the "sysctl_hung_task_warnings" global
> > limit.
> > 
> > Best Regards,
> 
> Hi Petr,
> 
> Thank you for the detailed breakdown and for taking the time to review the
> underlying issues. I appreciate the advice regarding the development
> cadence; taking a step back to consider broader feedback is certainly
> prudent, and I am grateful for your perspective.
> 
> Your proposal (a) is indeed elegant. Resetting sysctl_hung_task_warnings
> when check_hung_uninterruptible_tasks() detects zero blocked tasks provides
> a clean, deterministic way to recover from the "blind spot" regression
> without adding architectural complexity.
> 
> Regarding the blocker tracking, I take your point that a hash array is
> likely overkill for the common scenarios encountered in production. I
> suggest we combine your proposal (a) with a simplified version of (b):
> implementing a fixed-size array (e.g., array[10]) to track unique blockers.
> 
> This hybrid approach would function as follows:
> 
>     1.  When khungtaskd identifies a hung task, it compares the blocker
>         against the array. If the blocker is already tracked, we suppress
>         the warning, which keeps our sysctl_hung_task_warnings budget
>         intact. If the blocker is new, we print the warning, add it to the
>         array, and decrement the budget.
> 
>     2.  As you suggested, we reset the budget and clear the array[10] once
>         the hang resolves, accompanied by your proposed recovery message in
>         the ring buffer.
> 
> This seems to offer the best of both worlds: it provides the necessary
> filtering to prevent log bloat during lock storms while ensuring the system
> remains observable once the contention clears. It also adheres to the
> principle of addressing the primary regression while keeping the
> implementation overhead minimal.

I agree.

> Regarding the size of the array, I initially considered a size of 10 to
> keep the footprint minimal. However, I am happy to increase this to 32 to
> ensure we have sufficient coverage for complex multi-lock contention
> scenarios without incurring significant cache overhead. Does 32 strike you
> as a reasonable middle ground, or would you prefer I stick to a smaller
> fixed size?

32 sounds like a good compromise. We should print a warning when
it gets full. Let's see how it works in practice.

> I am happy to draft a v5 patch that combines these elements, should you
> agree that this remains a sensible direction.

Sounds reasonable from my POV.

Best Regards,
Petr