From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C8E14A13B7 for ; Thu, 2 Jul 2026 13:19:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.51 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782998366; cv=none; b=TNQALjNHiv7knCyLiB/taMGxv57GfeAankA4tpvP0BQLtm+FY6m8hBp2pdFkRm+SU/l8+Yki6TkBLB687FUuPX9sH94xawtHDiOU7mq0IAlnArhHTM3W70ocy42zRYNrKKWtcj9WGlYIrguNxDiupSJYnlLXSiwCWwgBxKoJ8bE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782998366; c=relaxed/simple; bh=3aE9fTCvZIeZpPMAoMwA5DABl7AlYYFFZv4DckF6gSo=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=g9r2e+6noUr6smzzdk553RaRb1XxUyRXzWxeG6qGFOi7ps2MriR5Z9swIgz9zlG57ILm9yB6c/Oa2eV1F/PuYPoEKqDdARP/BqYz9jKvG0PaFotMHcHEsYzvR0XUZO1vhLjJnzH0IOgO7sjdI/4kSqYrim046ASWUqgSAh9LesM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b=HZf4NhKC; arc=none smtp.client-ip=209.85.128.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b="HZf4NhKC" Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-493bb510ce4so13474255e9.1 for ; Thu, 02 Jul 2026 06:19:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1782998361; x=1783603161; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=wI13RXiwOm8vktGeozyj7mYwgm13SeoD9Eb4si66+oE=; b=HZf4NhKCJIFDVZQJGsec6oGadVm5gtw7E77BjUqbVtX9eZUC4hzMgs635lAFF7Hcj7 OEDcrrwpgCaQ174J0YJFn2CmC8P6ETYAEayAWIuQwNmUOThnGubb6uTsQTFZJjorCxyR 9LBvQxiN67WJ3niVRG17ow3NKVlIH3M1xuWPd+2WJGXWHaRtFcaDQV0EO3UcuV2DxG22 9Z0GZqMn8flScFA8qPdAfYoih7h73Q/OXhmZCwMkATDmFi+XBjjLk2Xb7Uu0z4qOVf4j PSuZ9Ru03qkLycYHqo8VM2fiOALWyJK0upsirK9ubdC8IHFLrX3QG4BnGf8cpEJtbfmE Cjug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782998361; x=1783603161; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wI13RXiwOm8vktGeozyj7mYwgm13SeoD9Eb4si66+oE=; b=j9f3V712heWcHYUkaFzP1ogw8vCC3KJivglcKgrayIdzyeYJB7lMshZfM9d+ICirZc L+j/DtFp2YDyllTXI6/ZsoWnysWz7FJtNLpOaCcyxVsdiT68oIM8R7fK5BNnMJvnC/KA 17w5zwUPTMC1/plkNZm+t9eVad+GziT58T3yX28ViurvO4Juz98heAczS9A6uHimRrAm k/UcPgUWo1nhInPQLUmPEswYxggvI6RO0QRFeMP/6UcR+FMSSbU5zWjagqZJnPt/zcLm 7Nc745/qrjkLKeGrvpTalpRqDP6/hpv26QIlZpXpRh1nkTCYApv3b0hGCknwgdgR92Xx 1aog== X-Forwarded-Encrypted: i=1; AFNElJ8ydwIR1Nq0d5f/7UhLJFPhvdK4PMITM0l1cvP+u3/6QCfINcNCJJQBA9sOI8ZhIt+cpjXh90qOkhTapWE=@vger.kernel.org X-Gm-Message-State: AOJu0Yyursnk8TUCQvS4DonUwwNdTO3/Nz926gCd99/KadeuR/+UY8ZA XEn8aYi05zkMBLQp/XQq5vej3E/AFESOXrKWieLRSQHhsfRj767eRbgo5ZzAGZ3i5CDJS3tl1zw RruEowuc= X-Gm-Gg: AfdE7cmgl+2t17ve4ZWgwsxLJKIpGsWOzIcFOVmsKKyTR7MIzs7ROBGyRNCRo+tZk+I v0mqyOUB5fmpQnFyk91JzU5PBF3XTdvrfELzP3mOrxpxUZVZmGffVEJxTJ13WcvRxFNuMibEc8C wTXM4CekLfJUcibYE6oNEeiGTAReMNSh9Dv21zKfKvMKtAuvPJfJGPHCiSdsf7wepBpsXY/7Dc2 EWqL2SphC+QMd+ioAdWNVZnk9xub3N8uSCh7BEcDrW9uix/pXyemikQm4GYANvWvNghEdGkZCgf e9Zt57C8h5hUVzU507BGZtijn96mSp1qo50CggbcgJkrpXVhAyZiZYyFyWH0w35BEoOfOO5Gflo XUyLH+eKeQmdx4GwQQRKq5vCI52REgCFgJ0FEQC3vJl6NCEvx1vg6MVsDD59gHSybSwOOAPmd6K Nm65qaHz7BD566i0s= X-Received: by 2002:a05:600c:6308:b0:493:b915:4a6f with SMTP id 5b1f17b1804b1-493c2b38138mr90068295e9.8.1782998361418; Thu, 02 Jul 2026 06:19:21 -0700 (PDT) Received: from pathway.suse.cz ([176.114.240.130]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-493c6368f89sm44830325e9.8.2026.07.02.06.19.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Jul 2026 06:19:20 -0700 (PDT) Date: Thu, 2 Jul 2026 15:19:18 +0200 From: Petr Mladek To: Aaron Tomlin Cc: Lance Yang , akpm@linux-foundation.org, mhiramat@kernel.org, linux-kernel@vger.kernel.org, david.laight.linux@gmail.com, neelx@suse.com, sean@ashe.io, chjohnst@gmail.com, steve@abita.co, mproche@gmail.com, nick.lange@gmail.com Subject: Re: [PATCH v4] hung_task: Deduplicate identical hang reports using explicit blocker tracking Message-ID: References: <20260627205733.90983-1-atomlin@atomlin.com> <20260628044750.32878-1-lance.yang@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Sun 2026-06-28 16:56:39, Aaron Tomlin wrote: > On Sun, Jun 28, 2026 at 12:47:50PM +0800, Lance Yang wrote: > To step back and address your question directly regarding the real-world > problem this patch aims to solve: Thanks a lot for slowing down. It allows people to think more about the problem and get feedback from more poeple. And it reduces the risk of burn out of maintainers and reviewers. > In large-scale, multi-tenant, production environments, lock contention is a > frequent reality. When a core resource (e.g., a heavily contended rwsem or > mutex) blocks, it does not just hang one task; it causes a cascading > failure that halts hundreds of tasks simultaneously. > > When khungtaskd runs its scan during such an outage, it often reports > identical stack traces into the kernel ring buffer, which is not entirely > useful. > > The global sysctl_hung_task_warnings budget is instantly exhausted by a > single lock storm. Consequently, the kernel is left entirely blind to > subsequent, completely unrelated deadlocks occurring elsewhere in the > system hours later. > > The changes introduced to date, moving away from the heuristic wchan > approach to a more deterministic t->blocker tracking as per Petr's > feedback, were an attempt to solve this without introducing complex > heuristics or dangerous blind spots. I would split this into two problems: 1. A single lock contention might trigger hung_report for many tasks waiting for the same lock. It bloats the kernel log and messages might even get lost. 2. The number of printed backtraces can be reduced by a global limit. But the limit silences the hung task detector and system administrators are blind once the limit is reached. IMHO, the global limit "sysctl_hung_task_warnings" has been introduced because of the 1st problem but it caused the 2nd problem. My proposal: ------------ a) We could Change the semantic of "sysctl_hung_task_warnings". It could newly limit the number of printed backtraces in a single hung-system situations. I mean to reset it when the check_hung_uninterruptible_tasks() does not detect any blocked tasks. We could even print a message in this case. Something like: if (atomic_long_read(&sysctl_hung_task_detect_count) && !this_round_count) { pr_err("INFO: Tasks are not blocked by sleeping locks any longer.\n"); atomic_long_set(&sysctl_hung_task_detect_count, 0); } This would keep it working for 1st problem and solve the 2nd problem. b) I like the check of task->blocker when it is available. But it depends on CONFIG_DETECT_HUNG_TASK_BLOCKER. Also the hash array looks like an overkill to me. I would replace the hash array with a simple array[10]. It should be enough in practice. Also it would be much easier to clear it when the hung situation has gone. That said, I am not sure if it is worth it. c) Also storing the info about printed backtraces into struct task_struct is interesting idea. But again, I am not sure if it is worth it. My opinion: ----------- I would start with a). It is trivial. It solves the regression caused by the current global limit. And the message about that the hung-situation has been resolved is useful. So it looks like win-win solution. I would do b) and/or c) only when a) is not enough in practice. That said: ---------- IMHO, CONFIG_DETECT_HUNG_TASK_BLOCKER is a rather cheap feature. I believe that the overhead is small especially when we are talking about sleeping locks. It is even enabled by default. Adding the filtering by the blocker might be more effective in practice than the "sysctl_hung_task_warnings" global limit. Best Regards, Petr