From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out-184.mta0.migadu.com (out-184.mta0.migadu.com [91.218.175.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1B372D5923 for ; Tue, 21 Jul 2026 05:08:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.184 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784610525; cv=none; b=AMIsKN3gey6yZDFGKvcDrzCXQ02sOlXOLfMs3vGQm+XAuQDjbXX8PB6n0XvoTbwjZO0PN0kah05zbuQpcT6/kjXG0Wg3H5B/z8MRAaZC/MRN801peZ2djYSx5UWenwN6pf2jaHBmwpj/L14xn7FgrBEVlEZT/CwL/QDgyplnbEE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784610525; c=relaxed/simple; bh=6YwjNR7yixa78+l2Q7sdfu42GfPhUu3ot6WjV4XEyBw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=a1PjKe6+6Fnp2/Ve7sfpm5wuySGGD1kfkXs7uBhaUzzTeMa893eVbXy+Zj7YJKoYFZdhCCVPu7TEXYqqVaWK+lxOC/RujYsCWL85g3Ug05tVlwOXyItx0ThJ9p4xeLZC2WarzGZbnh6kP8iOaH2LQbUBwxWup70ypASEPI4kiVQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=wX4qx7R/; arc=none smtp.client-ip=91.218.175.184 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="wX4qx7R/" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1784610520; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nflhwnUQAg0E7fd61CEhDh1XS1slsU/TXmqdQGsMbnM=; b=wX4qx7R/zArBqrSha57Iabuk5ybO+P1NGbfF402/xEJ1iczgMclDGx4wltJGEV/wjYx0L1 uxtHDDgOQ0bZVTl4wOITHCCTrgsln3jiCiI2RYRm9AOnNU9PU7Cb5XY0d5vx4iV/Nm6uQx o5Hft1JJwpELj3Anc8GGp/lT8FPXBuI= From: Lance Yang To: atomlin@atomlin.com, akpm@linux-foundation.org, pmladek@suse.com Cc: lance.yang@linux.dev, mhiramat@kernel.org, linux-kernel@vger.kernel.org, david.laight.linux@gmail.com, neelx@suse.com, sean@ashe.io, chjohnst@gmail.com, steve@abita.co, mproche@gmail.com, nick.lange@gmail.com Subject: Re: [PATCH v6] hung_task: Deduplicate identical hang reports using explicit blocker tracking Date: Tue, 21 Jul 2026 13:08:32 +0800 Message-Id: <20260721050832.68899-1-lance.yang@linux.dev> In-Reply-To: <20260719161305.428947-1-atomlin@atomlin.com> References: <20260719161305.428947-1-atomlin@atomlin.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT On Sun, Jul 19, 2026 at 12:13:05PM -0400, Aaron Tomlin wrote: >Currently, during severe lock contention, multiple tasks can hang while >waiting on the exact same resource. The khungtaskd kthread >indiscriminately reports every single instance with a stack trace. >This can roll the kernel ring buffer and prematurely exhaust the >kernel.hung_task_warnings budget. Consequently, the kernel is left >entirely blind to subsequent, unrelated deadlocks. > >To preserve the warning budget and ring buffer without sacrificing >observability, this patch introduces a two-tier deduplication mechanism: > > 1. Introduces a hung_task_reported in task_struct. If a task remains > hung across multiple check intervals, khungtaskd suppresses > redundant stack traces for that specific task until it makes > progress (verified via context switch counters). Furthermore, it > is packed into an existing compiler alignment hole, consuming > zero additional memory > > 2. For tasks blocked on a lock, we leverage the > CONFIG_DETECT_HUNG_TASK_BLOCKER infrastructure. By tracking the > exact memory address of the blocker lock in a persistent array of > size 32, we deduplicate warning reports for tasks waiting on the > same resource across check intervals. If the blocker is already > tracked, the warning is suppressed and the warning budget is > preserved > > 3. For duplicate tasks, we still print the single-line "INFO: task > ..." message and trigger tracepoint trace_sched_process_hang(). > It merely skips calling sched_show_task() and > debug_show_blocker(), printing a concise suppression notice > instead > > 4. Once the hang resolves (i.e., no hung tasks are detected in a > check round), the budget is restored to the value captured at the > start of the hang, and the blocker array is cleared, accompanied > by a recovery message in the ring buffer > >Signed-off-by: Aaron Tomlin >--- Sorry for the late reply. Still, v6 looks far too complicated ... and touches too much core code for the problem you're trying to address ... Looked at this again ... Petr's earlier breakdown[1] makes sense. Both problems are real: " I would split this into two problems: 1. A single lock contention might trigger hung_report for many tasks waiting for the same lock. It bloats the kernel log and messages might even get lost. 2. The number of printed backtraces can be reduced by a global limit. But the limit silences the hung task detector and system administrators are blind once the limit is reached. " IMO, that's the tradeoff: keep log noise down without leaving users blind for good once budget runs out ... Still not sold on dedup, though. Sure, duplicate reports happen, but how often, really? Doesn't seem common enough to need this much code :) If we really want to handle this in kernel, how about a simple mode switch for hung_task_warnings? 0 = keep the current global budget 1 = allow up to hung_task_warnings reports in each scan ---8<--- diff --git a/kernel/hung_task.c b/kernel/hung_task.c index 6fcc94ce4ca9..4ec2937a98e6 100644 --- a/kernel/hung_task.c +++ b/kernel/hung_task.c @@ -58,6 +58,7 @@ unsigned long __read_mostly sysctl_hung_task_timeout_secs = CONFIG_DEFAULT_HUNG_ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs; static int __read_mostly sysctl_hung_task_warnings = 10; +static int __read_mostly sysctl_hung_task_warnings_mode; static int __read_mostly did_panic; static bool hung_task_call_panic; @@ -228,12 +229,13 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti * @t: Pointer to the detected hung task. * @timeout: Timeout threshold for detecting hung tasks * @this_round_count: Count of hung tasks detected in the current iteration + * @warning_budget: Warning budget for reporting hung tasks * * Print structured information about the specified hung task, if warnings * are enabled or if the panic batch threshold is exceeded. */ static void hung_task_info(struct task_struct *t, unsigned long timeout, - unsigned long this_round_count) + unsigned long this_round_count, int *warning_budget) { trace_sched_process_hang(t); @@ -247,9 +249,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain * accordingly */ - if (sysctl_hung_task_warnings || hung_task_call_panic) { - if (sysctl_hung_task_warnings > 0) - sysctl_hung_task_warnings--; + if (*warning_budget || hung_task_call_panic) { + if (*warning_budget > 0) + (*warning_budget)--; pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n", t->comm, t->pid, t->in_iowait ? " in I/O wait" : "", (jiffies - t->last_switch_time) / HZ); @@ -264,8 +266,8 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, sched_show_task(t); debug_show_blocker(t, timeout); - if (!sysctl_hung_task_warnings) - pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n"); + if (!*warning_budget) + pr_info("Hung task warning budget exhausted, see sysctl kernel.hung_task_warnings\n"); } touch_nmi_watchdog(); @@ -305,6 +307,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) struct task_struct *g, *t; unsigned long this_round_count; int need_warning = sysctl_hung_task_warnings; + int warnings_left = need_warning; + int *warning_budget = sysctl_hung_task_warnings_mode ? + &warnings_left : &sysctl_hung_task_warnings; unsigned long si_mask = hung_task_si_mask; /* @@ -334,7 +339,8 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) */ atomic_long_inc(&sysctl_hung_task_detect_count); this_round_count++; - hung_task_info(t, timeout, this_round_count); + hung_task_info(t, timeout, this_round_count, + warning_budget); } } unlock: @@ -483,6 +489,15 @@ static const struct ctl_table hung_task_sysctls[] = { .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_NEG_ONE, }, + { + .procname = "hung_task_warnings_mode", + .data = &sysctl_hung_task_warnings_mode, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, { .procname = "hung_task_detect_count", .maxlen = sizeof(unsigned long), --- Per-scan mode avoids the permanent blind spot, while still letting users pick whichever behavior fits their setup. Keeps the change local, too. If that turns out not to be enough, sure, revisit dedup then. That's the direction I'd support :D [1] https://lore.kernel.org/lkml/akZlVgbImH6Jqy55@pathway.suse.cz/ Thanks, Lance