From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A4D422417D9 for ; Tue, 21 Jul 2026 13:28:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.43 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784640534; cv=none; b=ZYVVps4bAsUK5RSYlpYTmuR/e/iiOLeHd0BeH7Pd4huGZQ6dnpy4jDxxga5GPlGUJvxbjmx1r8wvb6iYadIo/XmavhqecTfmOamNX1zarglkvFyMNVVwfJZnywNyptvQyMVx7nRfl5MT++8UOzvFs0h0rZ4QisOgRKrOgkctxUw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784640534; c=relaxed/simple; bh=YLyKZwi+dMqqgD9rl5SP5C4ng9L2w+dJv/9ape8A7zU=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Q8QEEX4BEUzUMvI3IYwXliHRUaLkRRpNYT6CEw7+xmDnHJ2F+ThLk3z0o/sjTHpxb39zZ4D3+jkEoMAU3ScLmX+nROOFFgrH0jmRLN2KszbILch6+1rrNO18SdtogmQxTBCdXiZK6efBiwf5/7i4xyEGuBQ6aRSZDWfMY5N1cuE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b=R7oD726N; arc=none smtp.client-ip=209.85.128.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b="R7oD726N" Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-4953e04ef16so54617155e9.2 for ; Tue, 21 Jul 2026 06:28:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1784640531; x=1785245331; darn=vger.kernel.org; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to:content-type; bh=tfOjPV47+Td2HI7eRvAmG7mMAQeQDqKDXPPMM81+3BY=; b=R7oD726N8Cr8fkmO4Xc/YG1SaS6SbXUs3b4V8Q6fkUkDG8STJoKGKHv9SVFvSVsk1o eyxrR555RuB6NDT5KgIdEyqQrGwamfyj9LfjuTU/7bBTu8YtQiwg5Be3YemT938NSfvD g+eL6C0h+wEKatjXuWFKtL0Zw7lSrDPuuqWkoD3mr64cfEYwt4FNOvYVpf2dSfs6VsRy nqahAgkbB047vmT3WGGhPautjZ27yhZKRC1erWQlesnQk4wvf2Cku2atV2wmenIbzbhh IfjmurtvfPCqzZAbHoqlZSut1vOK9uBqABwanbeG1iMGdnPwjNJ1Fss6sLfds26yptDh ngXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784640531; x=1785245331; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=tfOjPV47+Td2HI7eRvAmG7mMAQeQDqKDXPPMM81+3BY=; b=PSblnrZbCUSfSoTRN5bU9uoU+ZXbKKF4Pc3zvN4C8LIGXvB4378YDQl/dxcemTvJSs w+8hY2fI05Sop9nRAg5rOi+AiYQtJacME3IQnBcEJ1r9dhuyQdt1hdUY9pImjkmy+dAw /OgaeUd9QY41EiIGNwX3pwvHtJsdoW/lKDr3gsWIlqSgD2iuWmsBgnwpnrZsMGoo9t0O LWF1EUBjPg55fV0AEJEG7eIRRks0kBVJUWPzLq83Mvj6uuy9Rv9DLmVXGsMgyENdDCSn X8LAKEvhxZq2qKSrM+QpWLorv9Qus2kDEL8hYPAaGyvzfLmbZ9+GELUuL7Jk1inM23rp 3E1Q== X-Forwarded-Encrypted: i=1; AHgh+RoSMRT9x3nBaEbJQwsDph04v5TTsL6y14uy+jeezQZAMpw9G5mIMeVyefr6DZyKZafpxuZawmFcj/cDxWE=@vger.kernel.org X-Gm-Message-State: AOJu0YyVn+Os/kJyOEFJldG6SGsRF62NP/mJ8ZzWZf8D0kxFBE43TkxB lrdg1eTiMQioYqiSPid42oGU9d6oQpyYwjpCoYMJ4Chh1sQa0ijlUEghVeHQJfdVcGs= X-Gm-Gg: AfdE7cmEZ5SwNzHIEqV/Wn+JBnae3CwFDN311XWlhxSR5D4hXHHwehZrrrvIPet+z8F BJ6rjcFNz/tUHvVPcOoQXO3pqeye2DSLCZ8MfiJR7L0+Dv+80HC2ogXFbOxyrIInwpdCz7Y5qql qW/3Qh7GkgyNYiy36+cRo8+Miqd7zmqEqtVvd8Hxx1Fnfv1yKd6tSwJTq1x4FthVjxJ9Hvqc4R2 ZQFFzzAk7MQ/x16W8TUR12WTnQhxeVcmrvXAEYTj2GHm8ZOThoZN1BByuJnzwhhFo02U/Akf1Hg rK5420m00qE9DQ/iX7Ep9zPtNxQ+NSeHXK/VvqMJOhTx3O475xtpiq4Tu/aUMvDJPtrWJR/0q7Q M9gML9PI6sbI1KCQUT+sStYrOD1yFNizUnwqbKnzX5/GhQqVXb7pBDNOUAGWrsmddhRjb0WwNVX U+c6Ox X-Received: by 2002:a05:600c:4685:b0:495:4a77:1327 with SMTP id 5b1f17b1804b1-4954a771521mr220507855e9.33.1784640530674; Tue, 21 Jul 2026 06:28:50 -0700 (PDT) Received: from pathway.suse.cz ([176.114.240.130]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-47f63eeb360sm40465278f8f.34.2026.07.21.06.28.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jul 2026 06:28:50 -0700 (PDT) Date: Tue, 21 Jul 2026 15:28:48 +0200 From: Petr Mladek To: Lance Yang Cc: atomlin@atomlin.com, akpm@linux-foundation.org, mhiramat@kernel.org, linux-kernel@vger.kernel.org, david.laight.linux@gmail.com, neelx@suse.com, sean@ashe.io, chjohnst@gmail.com, steve@abita.co, mproche@gmail.com, nick.lange@gmail.com Subject: Re: [PATCH v6] hung_task: Deduplicate identical hang reports using explicit blocker tracking Message-ID: References: <20260719161305.428947-1-atomlin@atomlin.com> <20260721050832.68899-1-lance.yang@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260721050832.68899-1-lance.yang@linux.dev> On Tue 2026-07-21 13:08:32, Lance Yang wrote: > > On Sun, Jul 19, 2026 at 12:13:05PM -0400, Aaron Tomlin wrote: > >Currently, during severe lock contention, multiple tasks can hang while > >waiting on the exact same resource. The khungtaskd kthread > >indiscriminately reports every single instance with a stack trace. > >This can roll the kernel ring buffer and prematurely exhaust the > >kernel.hung_task_warnings budget. Consequently, the kernel is left > >entirely blind to subsequent, unrelated deadlocks. > > > >To preserve the warning budget and ring buffer without sacrificing > >observability, this patch introduces a two-tier deduplication mechanism: > > > > 1. Introduces a hung_task_reported in task_struct. If a task remains > > hung across multiple check intervals, khungtaskd suppresses > > redundant stack traces for that specific task until it makes > > progress (verified via context switch counters). Furthermore, it > > is packed into an existing compiler alignment hole, consuming > > zero additional memory > > > > 2. For tasks blocked on a lock, we leverage the > > CONFIG_DETECT_HUNG_TASK_BLOCKER infrastructure. By tracking the > > exact memory address of the blocker lock in a persistent array of > > size 32, we deduplicate warning reports for tasks waiting on the > > same resource across check intervals. If the blocker is already > > tracked, the warning is suppressed and the warning budget is > > preserved > > > > 3. For duplicate tasks, we still print the single-line "INFO: task > > ..." message and trigger tracepoint trace_sched_process_hang(). > > It merely skips calling sched_show_task() and > > debug_show_blocker(), printing a concise suppression notice > > instead > > > > 4. Once the hang resolves (i.e., no hung tasks are detected in a > > check round), the budget is restored to the value captured at the > > start of the hang, and the blocker array is cleared, accompanied > > by a recovery message in the ring buffer > > > >Signed-off-by: Aaron Tomlin > >--- > > Sorry for the late reply. Don't worry. IMHO, a reply within one week is perfectly fine. Sometimes even this is not possible. > Still, v6 looks far too complicated ... and touches too much core code > for the problem you're trying to address ... I agree that it looks too complicated. I see several problems: 1. Many changes are added in a single patch. I suggest to split it next time into more patches and add the features from the most important one, ... 2. The code seems to be a bit over-engineered. There are too many variables hung_task_blockers_count, hung_task_has_active, warnings_decremented, skip_show_task, scan_completed, did_report, seen_blockers_mask, ... I believe that it might be done an easier way. 3. Too much spaghetti code is added to the already large check_hung_uninterruptible_tasks() function. It might look better when the detection of the duplicated task is handled in a separate function, ... > Looked at this again ... Petr's earlier breakdown[1] makes sense. Both > problems are real: > > " > I would split this into two problems: > > 1. A single lock contention might trigger hung_report for many tasks > waiting for the same lock. It bloats the kernel log and messages > might even get lost. > > 2. The number of printed backtraces can be reduced by a global limit. > But the limit silences the hung task detector and system > administrators are blind once the limit is reached. > " > > IMO, that's the tradeoff: keep log noise down without leaving users > blind for good once budget runs out ... > > Still not sold on dedup, though. Sure, duplicate reports happen, but how > often, really? Doesn't seem common enough to need this much code :) > > If we really want to handle this in kernel, how about a simple mode > switch for hung_task_warnings? > > 0 = keep the current global budget > 1 = allow up to hung_task_warnings reports in each scan Honestly, I do not think that this is the right way to go. The more I think about it, the more I believe that the global budget is simply wrong. IMHO, we should do the following: 1. Reset the global budget when the situation is resolved and there is no hung task any longer. 2. The global budget should limit only printing all the details, especially backtraces. We should always print at least the task name, ... 3. We could prevent printing the same process again and again by adding "hung_task_reported" into struct task_struct. But then we should print some summary about how many hung tasks were found in the last round at least. So that we know that the problem persists. This should still be rather easy. But I am not sure if it is worth it. 4. We could add the filtering of the duplicated backtraces. But it has to be handled by some helper function. It looks like to most complicated part. I would personally skip it for now. We could always add it later when the above proposed changes are not enough to tune the kernel.hung_task_warnings value in the real life. I tried to implement the 1st and 2nd change using gemini just to see how it would look. And it looks good to me: >From 4843170a1317f2fd3bebc1efc63c387840a4ed0d Mon Sep 17 00:00:00 2001 From: Petr Mladek Date: Tue, 21 Jul 2026 14:09:33 +0200 Subject: [PATCH 1/2] hung_task: Reset warning budget when problem gets resolved sysctl_hung_task_warnings counts how many hung tasks are reported. The watchdog does not report anything once the limit is reached. Currently, this budget is decremented permanently, meaning the kernel is left blind to subsequent hung tasks even after the original issue resolves. Keep the global sysctl_hung_task_warnings intact, and instead decrement a copy (hung_task_warnings_printed) when warnings are printed. Reset the copy back to the configured sysctl_hung_task_warnings limit once the problem on the system gets resolved and check_hung_uninterruptible_tasks() detects no hung tasks in a check interval. Also keep the copy updated when the global sysctl_hung_task_warnings value is updated via sysctl, and update documentation to reflect the new behavior. Assisted-by: gemini-3.5-flash Signed-off-by: Petr Mladek --- Documentation/admin-guide/sysctl/kernel.rst | 6 ++-- kernel/hung_task.c | 34 ++++++++++++++++----- 2 files changed, 31 insertions(+), 9 deletions(-) diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst index c6994e55d141..eaa1d837c739 100644 --- a/Documentation/admin-guide/sysctl/kernel.rst +++ b/Documentation/admin-guide/sysctl/kernel.rst @@ -459,8 +459,10 @@ hung_task_warnings ================== The maximum number of warnings to report. During a check interval -if a hung task is detected, this value is decreased by 1. -When this value reaches 0, no more warnings will be reported. +if a hung task is detected, the internal warning budget is decreased by 1. +When this budget reaches 0, no more warnings will be reported. +The warning budget is reset back to the configured limit when the problem on the +system gets resolved and no hung task is found. This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled. -1: report an infinite number of warnings. diff --git a/kernel/hung_task.c b/kernel/hung_task.c index 6fcc94ce4ca9..4e1fb0db79d1 100644 --- a/kernel/hung_task.c +++ b/kernel/hung_task.c @@ -59,6 +59,8 @@ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs; static int __read_mostly sysctl_hung_task_warnings = 10; +static int hung_task_warnings_printed = 10; + static int __read_mostly did_panic; static bool hung_task_call_panic; @@ -247,9 +249,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain * accordingly */ - if (sysctl_hung_task_warnings || hung_task_call_panic) { - if (sysctl_hung_task_warnings > 0) - sysctl_hung_task_warnings--; + if (hung_task_warnings_printed || hung_task_call_panic) { + if (hung_task_warnings_printed > 0) + hung_task_warnings_printed--; pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n", t->comm, t->pid, t->in_iowait ? " in I/O wait" : "", (jiffies - t->last_switch_time) / HZ); @@ -264,7 +266,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, sched_show_task(t); debug_show_blocker(t, timeout); - if (!sysctl_hung_task_warnings) + if (!hung_task_warnings_printed) pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n"); } @@ -304,7 +306,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) unsigned long last_break = jiffies; struct task_struct *g, *t; unsigned long this_round_count; - int need_warning = sysctl_hung_task_warnings; + int need_warning = hung_task_warnings_printed; unsigned long si_mask = hung_task_si_mask; /* @@ -340,8 +342,10 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) unlock: rcu_read_unlock(); - if (!this_round_count) + if (!this_round_count) { + hung_task_warnings_printed = sysctl_hung_task_warnings; return; + } if (need_warning || hung_task_call_panic) { si_mask |= SYS_INFO_LOCKS; @@ -425,6 +429,22 @@ static int proc_dohung_task_timeout_secs(const struct ctl_table *table, int writ return ret; } +static int proc_dohung_task_warnings(const struct ctl_table *table, int write, + void *buffer, + size_t *lenp, loff_t *ppos) +{ + int ret; + + ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos); + + if (ret || !write) + return ret; + + hung_task_warnings_printed = sysctl_hung_task_warnings; + + return 0; +} + /* * This is needed for proc_doulongvec_minmax of sysctl_hung_task_timeout_secs * and hung_task_check_interval_secs @@ -480,7 +500,7 @@ static const struct ctl_table hung_task_sysctls[] = { .data = &sysctl_hung_task_warnings, .maxlen = sizeof(int), .mode = 0644, - .proc_handler = proc_dointvec_minmax, + .proc_handler = proc_dohung_task_warnings, .extra1 = SYSCTL_NEG_ONE, }, { -- 2.55.0 >From 50f69511d706484d9378c019f8cbaa3232555fa6 Mon Sep 17 00:00:00 2001 From: Petr Mladek Date: Tue, 21 Jul 2026 14:49:16 +0200 Subject: [PATCH 2/2] hung_task: Always print basic hung task info header Currently, hung_task_info() completely suppresses any warning messages once the warning budget is reached. This leaves the system completely blind to any future hung tasks, even though we only wanted to suppress heavy process stack dumps. Modify hung_task_info() to always print the basic single-line error message about a hung task, regardless of whether the warning budget has been exhausted. Restrict only the verbose details (such as stack dumps, taint/release status, and blockers) to when the warning budget is intact or panic is triggered. Additionally, update the notice printed upon warning exhaustion to clarify that future reports will only omit process details rather than being completely suppressed. Assisted-by: gemini-3.5-flash Signed-off-by: Petr Mladek --- kernel/hung_task.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/kernel/hung_task.c b/kernel/hung_task.c index 4e1fb0db79d1..6ebb3a87ac65 100644 --- a/kernel/hung_task.c +++ b/kernel/hung_task.c @@ -244,17 +244,19 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, hung_task_call_panic = true; } + /* Always print the blocked message */ + pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n", + t->comm, t->pid, t->in_iowait ? " in I/O wait" : "", + (jiffies - t->last_switch_time) / HZ); + /* * The given task did not get scheduled for more than * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain - * accordingly + * accordingly with full details if the budget is not exhausted. */ if (hung_task_warnings_printed || hung_task_call_panic) { if (hung_task_warnings_printed > 0) hung_task_warnings_printed--; - pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n", - t->comm, t->pid, t->in_iowait ? " in I/O wait" : "", - (jiffies - t->last_switch_time) / HZ); pr_err(" %s %s %.*s\n", print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, " "), @@ -267,7 +269,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, debug_show_blocker(t, timeout); if (!hung_task_warnings_printed) - pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n"); + pr_info("Future hung task reports won't print details about each process, see sysctl kernel.hung_task_warnings\n"); } touch_nmi_watchdog(); -- 2.55.0