From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6E2A3F4DC1 for ; Tue, 7 Jul 2026 12:43:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.45 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783428231; cv=none; b=Hxvj03wkMv0Uzltjybu3GZRDmtJ28q5Sypm4T1PrZJ31Da84So49pwp2QsZaTZl9D0v48jNmcAj0yLCIjncgHBUnKo2I/Wsnj1y1ZHQSU2XAl1+E1mnJ86QDAEHF/3RND5gupvU/RgR+yGAHpkE22Kz+A0iqkulXcjHix6zs2KE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783428231; c=relaxed/simple; bh=HQ7WkKblmFyNQeNTbVKGbAvnGCg8UBNZRT5KjgJNsOY=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Kqj2a7MDa3131xdJTJGgIQ3VRgDU3GweyhU7GkD2DsHzZPPpWN7ZCi8K59shHkoGaHELzLtKasQ1UIVhugiiI/lw3KQlvwuFn12Ni3EsDsoMfG3or45FqnHCBcVyIBMFJOxqtnTbPXXmJUsRC7hdyYxorzIHN8hWHRtRzsuXAbQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b=QsIdIx5h; arc=none smtp.client-ip=209.85.128.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b="QsIdIx5h" Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-493ae59eca6so29794465e9.1 for ; Tue, 07 Jul 2026 05:43:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1783428228; x=1784033028; darn=vger.kernel.org; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to:content-type; bh=f2g3lV+Yh6ixHsHtqbw4NQe4S0L6jBNP73zwbdRADo4=; b=QsIdIx5hRk/owqVcERQwoX0/wYOLaCrAQuqmQ/nWMgq+qIqmGMfw1IjmlXbYrZWeTB 7/wNyG1whAGkkDLMVLMCzJL2+r8FCaGC9iIJz69YPmYpelxYcWljaYdd6pKa4FmLmuRK We5BlAvC3+KB03+BzkoHGXakhvgsoPEy9mye4Sb5WgVNO6Z4ZWYLQImIXUqPeJwjtJU3 z07memWp/bGhsod5vCvHUURanh+BJblCf5IOA2eSIS2JGRm+dCUxXSWhBP4apLFLVCVK DCcui3v+AVg4eqXBiQ+ra3peSE1/TwnohIh6+qN5qzM5VsLe1Sz7jOE2j/U7GWa4oUKG uiiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783428228; x=1784033028; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=f2g3lV+Yh6ixHsHtqbw4NQe4S0L6jBNP73zwbdRADo4=; b=SDC8Ezbp+FlsW5vKT8VEIhhoVxbqSktUyXVOTfW2qFWXm7tD7ESIoe/z2Ad79ogW3/ f7z/rKefDCRURzda0NynSLKyZmYRtZfMt4tlaO7ThL7pXFrn4H6O8UMcTLXoIgFviY3s eyWT21ZOPLrPMAfIStmGG3YjTlr7aH9YlwBgo7hh9MgjmIC79xw4XnSCNipgUyQBtCM6 ikVCBlPPkgm0YOFSMOkIn3muWlyT7fdCQ6ErUtBwn6j11ahAyw4qypdUQ1i5pwZe5SMe 1PWI1aN9sEWBuXWHuUgZ/w04ltvJHVfJQ32XaX6XjGXYjuEeGYlC+3G0k5gI+yJvls2v UH/g== X-Forwarded-Encrypted: i=1; AHgh+Rqs1DKJ4O0vnRysf6w8sGqtWBYTii4TXAcTqtzbxSgLLLz6IEq8sudJayoyW6PgUUczuIAPNzEkhh+9umU=@vger.kernel.org X-Gm-Message-State: AOJu0YzM0Z19+QiIlE48Da+QVE0cqHxVxtym383ZfYDbAUHvJTBe2L6l P3pxm11RNAfeU4BoCmlErQaK54xa0kbpr5gHqsaizeiujyak6XRe0FL9LbL5c/RLBfQ= X-Gm-Gg: AfdE7cnBsYxhWYEeW5rZPmFEYa2Px4nD91fzxeIDViLajG2DiUnzFxT9aTSjoLa8Tdb CTeTvi0VE7feUCnbSVtqX2zkk0Zvr01qhZs/A8E1ihI0TMUrRfqZArHHAVo2ConUHqMh7wOR+/4 7EkwtI1ArDBBBfLX9wGR133AY3WsrUdlPmQgqFFR4GCHsRBLNybDdlAiVMPW3vIPZ3lBmBdC3am ZB5aKtpai+z0pIhI+biJypxm87kdosd64YMTNr74kU0oDDC86gFV7tDppteMIizOJm4rbX+Vwv+ iHxAWh2NiT+e7e2tjI2xCkyCdbFhnduLy/qirhYR1pLRrQu4uzUuogsM4+WSfm8Vx3m72f2IDHT c9tbVNYtKqODOaaercYzewtbAODwawrD9WAkhFnSs7TgonTAUQFFpWHrV0VHFCMlFNSUsQXe9t0 A+SpeS9pe81D60FoseNSjD+7bxGw== X-Received: by 2002:a05:600c:8b82:b0:493:bd2a:93bb with SMTP id 5b1f17b1804b1-493df038790mr58694475e9.3.1783428228117; Tue, 07 Jul 2026 05:43:48 -0700 (PDT) Received: from pathway.suse.cz ([176.114.240.130]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-47aa0960816sm32314934f8f.29.2026.07.07.05.43.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Jul 2026 05:43:47 -0700 (PDT) Date: Tue, 7 Jul 2026 14:43:45 +0200 From: Petr Mladek To: Aaron Tomlin Cc: Lance Yang , akpm@linux-foundation.org, mhiramat@kernel.org, linux-kernel@vger.kernel.org, david.laight.linux@gmail.com, neelx@suse.com, sean@ashe.io, chjohnst@gmail.com, steve@abita.co, mproche@gmail.com, nick.lange@gmail.com Subject: Re: [PATCH v4] hung_task: Deduplicate identical hang reports using explicit blocker tracking Message-ID: References: <20260627205733.90983-1-atomlin@atomlin.com> <20260628044750.32878-1-lance.yang@linux.dev> <3gccq2oqarpxwe7fgf7it5vxz5kqhvcqapv6pfw5yzufktwo2t@re25go4kajjb> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3gccq2oqarpxwe7fgf7it5vxz5kqhvcqapv6pfw5yzufktwo2t@re25go4kajjb> On Sat 2026-07-04 17:35:29, Aaron Tomlin wrote: > On Thu, Jul 02, 2026 at 03:19:18PM +0200, Petr Mladek wrote: > > On Sun 2026-06-28 16:56:39, Aaron Tomlin wrote: > > > On Sun, Jun 28, 2026 at 12:47:50PM +0800, Lance Yang wrote: > > > To step back and address your question directly regarding the real-world > > > problem this patch aims to solve: > > > > Thanks a lot for slowing down. It allows people to think more about > > the problem and get feedback from more poeple. And it reduces the risk > > of burn out of maintainers and reviewers. > > > > > In large-scale, multi-tenant, production environments, lock contention is a > > > frequent reality. When a core resource (e.g., a heavily contended rwsem or > > > mutex) blocks, it does not just hang one task; it causes a cascading > > > failure that halts hundreds of tasks simultaneously. > > > > > > When khungtaskd runs its scan during such an outage, it often reports > > > identical stack traces into the kernel ring buffer, which is not entirely > > > useful. > > > > > > The global sysctl_hung_task_warnings budget is instantly exhausted by a > > > single lock storm. Consequently, the kernel is left entirely blind to > > > subsequent, completely unrelated deadlocks occurring elsewhere in the > > > system hours later. > > > > > > The changes introduced to date, moving away from the heuristic wchan > > > approach to a more deterministic t->blocker tracking as per Petr's > > > feedback, were an attempt to solve this without introducing complex > > > heuristics or dangerous blind spots. > > > > I would split this into two problems: > > > > 1. A single lock contention might trigger hung_report for many tasks > > waiting for the same lock. It bloats the kernel log and messages > > might even get lost. > > > > 2. The number of printed backtraces can be reduced by a global limit. > > But the limit silences the hung task detector and system > > administrators are blind once the limit is reached. > > > > IMHO, the global limit "sysctl_hung_task_warnings" has been introduced > > because of the 1st problem but it caused the 2nd problem. > > > > My proposal: > > ------------ > > > > a) We could Change the semantic of "sysctl_hung_task_warnings". It could newly > > limit the number of printed backtraces in a single hung-system > > situations. I mean to reset it when the check_hung_uninterruptible_tasks() > > does not detect any blocked tasks. > > > > We could even print a message in this case. Something like: > > > > if (atomic_long_read(&sysctl_hung_task_detect_count) && !this_round_count) { > > pr_err("INFO: Tasks are not blocked by sleeping locks any longer.\n"); > > atomic_long_set(&sysctl_hung_task_detect_count, 0); > > } > > > > This would keep it working for 1st problem and solve the 2nd problem. > > > > > > b) I like the check of task->blocker when it is available. But it > > depends on CONFIG_DETECT_HUNG_TASK_BLOCKER. Also the hash array > > looks like an overkill to me. > > > > I would replace the hash array with a simple array[10]. It > > should be enough in practice. Also it would be much easier > > to clear it when the hung situation has gone. > > > > That said, I am not sure if it is worth it. > > > > > > c) Also storing the info about printed backtraces into struct > > task_struct is interesting idea. > > > > But again, I am not sure if it is worth it. > > > > > > My opinion: > > ----------- > > > > I would start with a). It is trivial. It solves the regression caused > > by the current global limit. And the message about that > > the hung-situation has been resolved is useful. So it > > looks like win-win solution. > > > > I would do b) and/or c) only when a) is not enough in practice. > > > > That said: > > ---------- > > > > IMHO, CONFIG_DETECT_HUNG_TASK_BLOCKER is a rather cheap feature. > > I believe that the overhead is small especially when we are > > talking about sleeping locks. It is even enabled by default. > > > > Adding the filtering by the blocker might be more effective > > in practice than the "sysctl_hung_task_warnings" global > > limit. > > > > Best Regards, > > Hi Petr, > > Thank you for the detailed breakdown and for taking the time to review the > underlying issues. I appreciate the advice regarding the development > cadence; taking a step back to consider broader feedback is certainly > prudent, and I am grateful for your perspective. > > Your proposal (a) is indeed elegant. Resetting sysctl_hung_task_warnings > when check_hung_uninterruptible_tasks() detects zero blocked tasks provides > a clean, deterministic way to recover from the "blind spot" regression > without adding architectural complexity. > > Regarding the blocker tracking, I take your point that a hash array is > likely overkill for the common scenarios encountered in production. I > suggest we combine your proposal (a) with a simplified version of (b): > implementing a fixed-size array (e.g., array[10]) to track unique blockers. > > This hybrid approach would function as follows: > > 1. When khungtaskd identifies a hung task, it compares the blocker > against the array. If the blocker is already tracked, we suppress > the warning, which keeps our sysctl_hung_task_warnings budget > intact. If the blocker is new, we print the warning, add it to the > array, and decrement the budget. > > 2. As you suggested, we reset the budget and clear the array[10] once > the hang resolves, accompanied by your proposed recovery message in > the ring buffer. > > This seems to offer the best of both worlds: it provides the necessary > filtering to prevent log bloat during lock storms while ensuring the system > remains observable once the contention clears. It also adheres to the > principle of addressing the primary regression while keeping the > implementation overhead minimal. I agree. > Regarding the size of the array, I initially considered a size of 10 to > keep the footprint minimal. However, I am happy to increase this to 32 to > ensure we have sufficient coverage for complex multi-lock contention > scenarios without incurring significant cache overhead. Does 32 strike you > as a reasonable middle ground, or would you prefer I stick to a smaller > fixed size? 32 sounds like a good compromise. We should print a warning when it gets full. Let's see how it works in practice. > I am happy to draft a v5 patch that combines these elements, should you > agree that this remains a sensible direction. Sounds reasonable from my POV. Best Regards, Petr