* [PATCH 0/2] hung_task: Improve warning budget handling and task reporting
@ 2026-08-04 0:15 Aaron Tomlin
2026-08-04 0:15 ` [PATCH 1/2] hung_task: Reset warning budget when problem gets resolved Aaron Tomlin
2026-08-04 0:15 ` [PATCH 2/2] hung_task: Always print basic hung task info header Aaron Tomlin
0 siblings, 2 replies; 8+ messages in thread
From: Aaron Tomlin @ 2026-08-04 0:15 UTC (permalink / raw)
To: akpm, lance.yang, mhiramat, pmladek
Cc: linux-kernel, david.laight.linux, atomlin, neelx, sean, chjohnst,
steve, mproche, nick.lange
The hung_task watchdog detects tasks stuck in uninterruptible (D) state for
longer than CONFIG_DEFAULT_HUNG_TASK_TIMEOUT seconds. To prevent log spam
during system spikes, sysctl_hung_task_warnings enforces a budget on the
number of logged warnings.
However, the current implementation has two major limitations:
1. Permanent exhaustion of warning budget
sysctl_hung_task_warnings is decremented directly when printing
warnings. Once this budget hits zero, no further warnings are
reported until an administrator manually updates the sysctl value or
reboots the system. Consequently, a single temporary hang episode
permanently blinds the kernel watchdog to any subsequent hung tasks
after system recovery.
2. Total log suppression when budget is exhausted
Once the warning budget reaches zero, hung_task_info() completely
suppresses all output, including the basic single-line alert. While
suppressing verbose stack dumps and lock debugging is desirable to
prevent dmesg flooding, hiding basic task alerts leaves
administrators entirely unaware that tasks are hanging.
This patch series resolves both limitations by decoupling the configured
warning budget from the runtime warning counter, automatically resetting
the budget when the system recovers, and keeping basic single-line hung
task alerts visible.
Patch 1 decouples the user-configured limit sysctl_hung_task_warnings from
the runtime counter hung_task_warnings_printed, automatically resetting
hung_task_warnings_printed back to the configured limit whenever a check
interval passes with zero hung tasks detected. The runtime counter is also
kept synchronized whenever the sysctl parameter is modified, and the
corresponding sysctl documentation
(Documentation/admin-guide/sysctl/kernel.rst) is updated to reflect this
auto-reset behaviour.
Patch 2 ensures that the basic single-line pr_err("INFO: task %s:%d
blocked...") header is always logged regardless of warning budget
exhaustion, while restricting warning budget enforcement solely to verbose
diagnostics such as process stack dumps, taint and release information, and
lock blocker details. Additionally, it updates the warning exhaustion log
message to clarify to administrators that future reports will only omit
detailed process dumps rather than being completely suppressed.
Petr Mladek (2):
hung_task: Reset warning budget when problem gets resolved
hung_task: Always print basic hung task info header
Documentation/admin-guide/sysctl/kernel.rst | 6 ++-
kernel/hung_task.c | 46 +++++++++++++++------
2 files changed, 38 insertions(+), 14 deletions(-)
--
2.55.0
^ permalink raw reply [flat|nested] 8+ messages in thread* [PATCH 1/2] hung_task: Reset warning budget when problem gets resolved 2026-08-04 0:15 [PATCH 0/2] hung_task: Improve warning budget handling and task reporting Aaron Tomlin @ 2026-08-04 0:15 ` Aaron Tomlin 2026-08-04 2:24 ` Lance Yang 2026-08-04 3:18 ` Lance Yang 2026-08-04 0:15 ` [PATCH 2/2] hung_task: Always print basic hung task info header Aaron Tomlin 1 sibling, 2 replies; 8+ messages in thread From: Aaron Tomlin @ 2026-08-04 0:15 UTC (permalink / raw) To: akpm, lance.yang, mhiramat, pmladek Cc: linux-kernel, david.laight.linux, atomlin, neelx, sean, chjohnst, steve, mproche, nick.lange From: Petr Mladek <pmladek@suse.com> sysctl_hung_task_warnings counts how many hung tasks are reported. The watchdog does not report anything once the limit is reached. Currently, this budget is decremented permanently, meaning the kernel is left blind to subsequent hung tasks even after the original issue resolves. Keep the global sysctl_hung_task_warnings intact, and instead decrement a copy (hung_task_warnings_printed) when warnings are printed. Reset the copy back to the configured sysctl_hung_task_warnings limit once the problem on the system gets resolved and check_hung_uninterruptible_tasks() detects no hung tasks in a check interval. Also keep the copy updated when the global sysctl_hung_task_warnings value is updated via sysctl, and update documentation to reflect the new behavior. Assisted-by: gemini-3.5-flash Signed-off-by: Petr Mladek <pmladek@suse.com> --- Documentation/admin-guide/sysctl/kernel.rst | 6 ++-- kernel/hung_task.c | 34 ++++++++++++++++----- 2 files changed, 31 insertions(+), 9 deletions(-) diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst index c6994e55d141..eaa1d837c739 100644 --- a/Documentation/admin-guide/sysctl/kernel.rst +++ b/Documentation/admin-guide/sysctl/kernel.rst @@ -459,8 +459,10 @@ hung_task_warnings ================== The maximum number of warnings to report. During a check interval -if a hung task is detected, this value is decreased by 1. -When this value reaches 0, no more warnings will be reported. +if a hung task is detected, the internal warning budget is decreased by 1. +When this budget reaches 0, no more warnings will be reported. +The warning budget is reset back to the configured limit when the problem on the +system gets resolved and no hung task is found. This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled. -1: report an infinite number of warnings. diff --git a/kernel/hung_task.c b/kernel/hung_task.c index 6fcc94ce4ca9..4e1fb0db79d1 100644 --- a/kernel/hung_task.c +++ b/kernel/hung_task.c @@ -59,6 +59,8 @@ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs; static int __read_mostly sysctl_hung_task_warnings = 10; +static int hung_task_warnings_printed = 10; + static int __read_mostly did_panic; static bool hung_task_call_panic; @@ -247,9 +249,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain * accordingly */ - if (sysctl_hung_task_warnings || hung_task_call_panic) { - if (sysctl_hung_task_warnings > 0) - sysctl_hung_task_warnings--; + if (hung_task_warnings_printed || hung_task_call_panic) { + if (hung_task_warnings_printed > 0) + hung_task_warnings_printed--; pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n", t->comm, t->pid, t->in_iowait ? " in I/O wait" : "", (jiffies - t->last_switch_time) / HZ); @@ -264,7 +266,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, sched_show_task(t); debug_show_blocker(t, timeout); - if (!sysctl_hung_task_warnings) + if (!hung_task_warnings_printed) pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n"); } @@ -304,7 +306,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) unsigned long last_break = jiffies; struct task_struct *g, *t; unsigned long this_round_count; - int need_warning = sysctl_hung_task_warnings; + int need_warning = hung_task_warnings_printed; unsigned long si_mask = hung_task_si_mask; /* @@ -340,8 +342,10 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) unlock: rcu_read_unlock(); - if (!this_round_count) + if (!this_round_count) { + hung_task_warnings_printed = sysctl_hung_task_warnings; return; + } if (need_warning || hung_task_call_panic) { si_mask |= SYS_INFO_LOCKS; @@ -425,6 +429,22 @@ static int proc_dohung_task_timeout_secs(const struct ctl_table *table, int writ return ret; } +static int proc_dohung_task_warnings(const struct ctl_table *table, int write, + void *buffer, + size_t *lenp, loff_t *ppos) +{ + int ret; + + ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos); + + if (ret || !write) + return ret; + + hung_task_warnings_printed = sysctl_hung_task_warnings; + + return 0; +} + /* * This is needed for proc_doulongvec_minmax of sysctl_hung_task_timeout_secs * and hung_task_check_interval_secs @@ -480,7 +500,7 @@ static const struct ctl_table hung_task_sysctls[] = { .data = &sysctl_hung_task_warnings, .maxlen = sizeof(int), .mode = 0644, - .proc_handler = proc_dointvec_minmax, + .proc_handler = proc_dohung_task_warnings, .extra1 = SYSCTL_NEG_ONE, }, { -- 2.55.0 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH 1/2] hung_task: Reset warning budget when problem gets resolved 2026-08-04 0:15 ` [PATCH 1/2] hung_task: Reset warning budget when problem gets resolved Aaron Tomlin @ 2026-08-04 2:24 ` Lance Yang 2026-08-04 14:19 ` Aaron Tomlin 2026-08-04 3:18 ` Lance Yang 1 sibling, 1 reply; 8+ messages in thread From: Lance Yang @ 2026-08-04 2:24 UTC (permalink / raw) To: Aaron Tomlin Cc: linux-kernel, david.laight.linux, neelx, sean, chjohnst, steve, mproche, nick.lange, akpm, mhiramat, pmladek On 2026/8/4 08:15, Aaron Tomlin wrote: > From: Petr Mladek <pmladek@suse.com> > > sysctl_hung_task_warnings counts how many hung tasks are reported. > The watchdog does not report anything once the limit is reached. > Currently, this budget is decremented permanently, meaning the kernel is > left blind to subsequent hung tasks even after the original issue resolves. > > Keep the global sysctl_hung_task_warnings intact, and instead decrement > a copy (hung_task_warnings_printed) when warnings are printed. Reset > the copy back to the configured sysctl_hung_task_warnings limit once the > problem on the system gets resolved and check_hung_uninterruptible_tasks() > detects no hung tasks in a check interval. > > Also keep the copy updated when the global sysctl_hung_task_warnings > value is updated via sysctl, and update documentation to reflect > the new behavior. > > Assisted-by: gemini-3.5-flash > Signed-off-by: Petr Mladek <pmladek@suse.com> Was a bit surprised not to see any tag from you here. I’d assumed you’d be taking this through testing and spinning v7, so I expected your SoB + Suggested-by Petr. Keeping both SoBs works too; either way, you’ve definitely earned yours here. Assisted-by: gemini-3.5-flash Suggested-by: Petr Mladek <pmladek@suse.com> Signed-off-by: Aaron Tomlin <atomlin@atomlin.com> Same goes for patch #02, btw :) Thanks, Lance ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/2] hung_task: Reset warning budget when problem gets resolved 2026-08-04 2:24 ` Lance Yang @ 2026-08-04 14:19 ` Aaron Tomlin 0 siblings, 0 replies; 8+ messages in thread From: Aaron Tomlin @ 2026-08-04 14:19 UTC (permalink / raw) To: Lance Yang Cc: linux-kernel, david.laight.linux, neelx, sean, chjohnst, steve, mproche, nick.lange, akpm, mhiramat, pmladek [-- Attachment #1: Type: text/plain, Size: 1690 bytes --] On Tue, Aug 04, 2026 at 10:24:01AM +0800, Lance Yang wrote: > On 2026/8/4 08:15, Aaron Tomlin wrote: > > From: Petr Mladek <pmladek@suse.com> > > > > sysctl_hung_task_warnings counts how many hung tasks are reported. > > The watchdog does not report anything once the limit is reached. > > Currently, this budget is decremented permanently, meaning the kernel is > > left blind to subsequent hung tasks even after the original issue resolves. > > > > Keep the global sysctl_hung_task_warnings intact, and instead decrement > > a copy (hung_task_warnings_printed) when warnings are printed. Reset > > the copy back to the configured sysctl_hung_task_warnings limit once the > > problem on the system gets resolved and check_hung_uninterruptible_tasks() > > detects no hung tasks in a check interval. > > > > Also keep the copy updated when the global sysctl_hung_task_warnings > > value is updated via sysctl, and update documentation to reflect > > the new behavior. > > > > Assisted-by: gemini-3.5-flash > > Signed-off-by: Petr Mladek <pmladek@suse.com> > > Was a bit surprised not to see any tag from you here. I’d assumed you’d > be taking this through testing and spinning v7, so I expected your SoB + > Suggested-by Petr. > > Keeping both SoBs works too; either way, you’ve definitely earned yours > here. > > Assisted-by: gemini-3.5-flash > Suggested-by: Petr Mladek <pmladek@suse.com> > Signed-off-by: Aaron Tomlin <atomlin@atomlin.com> > > Same goes for patch #02, btw :) > > Thanks, Lance Hi Lance, Thank you. I will prepare a v7 with the above. Also, include your "Reviewed-by:" tag. Kind regards, -- Aaron Tomlin [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/2] hung_task: Reset warning budget when problem gets resolved 2026-08-04 0:15 ` [PATCH 1/2] hung_task: Reset warning budget when problem gets resolved Aaron Tomlin 2026-08-04 2:24 ` Lance Yang @ 2026-08-04 3:18 ` Lance Yang 2026-08-04 14:54 ` Aaron Tomlin 1 sibling, 1 reply; 8+ messages in thread From: Lance Yang @ 2026-08-04 3:18 UTC (permalink / raw) To: atomlin Cc: akpm, mhiramat, pmladek, linux-kernel, david.laight.linux, neelx, sean, chjohnst, steve, mproche, nick.lange, Lance Yang On Mon, Aug 03, 2026 at 08:15:41PM -0400, Aaron Tomlin wrote: >From: Petr Mladek <pmladek@suse.com> > >sysctl_hung_task_warnings counts how many hung tasks are reported. >The watchdog does not report anything once the limit is reached. >Currently, this budget is decremented permanently, meaning the kernel is >left blind to subsequent hung tasks even after the original issue resolves. > >Keep the global sysctl_hung_task_warnings intact, and instead decrement >a copy (hung_task_warnings_printed) when warnings are printed. Reset >the copy back to the configured sysctl_hung_task_warnings limit once the >problem on the system gets resolved and check_hung_uninterruptible_tasks() >detects no hung tasks in a check interval. > >Also keep the copy updated when the global sysctl_hung_task_warnings >value is updated via sysctl, and update documentation to reflect >the new behavior. > >Assisted-by: gemini-3.5-flash >Signed-off-by: Petr Mladek <pmladek@suse.com> >--- > Documentation/admin-guide/sysctl/kernel.rst | 6 ++-- > kernel/hung_task.c | 34 ++++++++++++++++----- > 2 files changed, 31 insertions(+), 9 deletions(-) > >diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst >index c6994e55d141..eaa1d837c739 100644 >--- a/Documentation/admin-guide/sysctl/kernel.rst >+++ b/Documentation/admin-guide/sysctl/kernel.rst >@@ -459,8 +459,10 @@ hung_task_warnings > ================== > > The maximum number of warnings to report. During a check interval >-if a hung task is detected, this value is decreased by 1. >-When this value reaches 0, no more warnings will be reported. >+if a hung task is detected, the internal warning budget is decreased by 1. >+When this budget reaches 0, no more warnings will be reported. >+The warning budget is reset back to the configured limit when the problem on the >+system gets resolved and no hung task is found. Maybe just go with this? " The maximum number of warnings to report. During a check interval if a hung task is detected, the internal warning budget is decreased by 1. When this budget reaches 0, no more detailed warnings will be reported. The warning budget is reset to the configured limit when no hung task is found. " > This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled. > > -1: report an infinite number of warnings. >diff --git a/kernel/hung_task.c b/kernel/hung_task.c >index 6fcc94ce4ca9..4e1fb0db79d1 100644 >--- a/kernel/hung_task.c >+++ b/kernel/hung_task.c >@@ -59,6 +59,8 @@ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs; > > static int __read_mostly sysctl_hung_task_warnings = 10; > >+static int hung_task_warnings_printed = 10; >+ > static int __read_mostly did_panic; > static bool hung_task_call_panic; > >@@ -247,9 +249,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, > * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain > * accordingly > */ >- if (sysctl_hung_task_warnings || hung_task_call_panic) { >- if (sysctl_hung_task_warnings > 0) >- sysctl_hung_task_warnings--; >+ if (hung_task_warnings_printed || hung_task_call_panic) { >+ if (hung_task_warnings_printed > 0) >+ hung_task_warnings_printed--; > pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n", > t->comm, t->pid, t->in_iowait ? " in I/O wait" : "", > (jiffies - t->last_switch_time) / HZ); >@@ -264,7 +266,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, > sched_show_task(t); > debug_show_blocker(t, timeout); > >- if (!sysctl_hung_task_warnings) >+ if (!hung_task_warnings_printed) > pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n"); > } > >@@ -304,7 +306,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) > unsigned long last_break = jiffies; > struct task_struct *g, *t; > unsigned long this_round_count; >- int need_warning = sysctl_hung_task_warnings; >+ int need_warning = hung_task_warnings_printed; > unsigned long si_mask = hung_task_si_mask; > > /* >@@ -340,8 +342,10 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) > unlock: > rcu_read_unlock(); > >- if (!this_round_count) >+ if (!this_round_count) { Hmm, should max_count, or more generally scan completion, be part of this condition too? Maybe not, since a limited scan is already allowed to miss tasks ... If max_count stopped us early, we’d reset the budget even though a hung task may still be later in the list ... I probably wouldn’t change it. Just wanted to throw the question out here. >+ hung_task_warnings_printed = sysctl_hung_task_warnings; > return; >+ } > > if (need_warning || hung_task_call_panic) { > si_mask |= SYS_INFO_LOCKS; >@@ -425,6 +429,22 @@ static int proc_dohung_task_timeout_secs(const struct ctl_table *table, int writ > return ret; > } > >+static int proc_dohung_task_warnings(const struct ctl_table *table, int write, >+ void *buffer, >+ size_t *lenp, loff_t *ppos) >+{ >+ int ret; >+ >+ ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos); >+ >+ if (ret || !write) >+ return ret; >+ >+ hung_task_warnings_printed = sysctl_hung_task_warnings; >+ >+ return 0; >+} >+ > /* > * This is needed for proc_doulongvec_minmax of sysctl_hung_task_timeout_secs > * and hung_task_check_interval_secs >@@ -480,7 +500,7 @@ static const struct ctl_table hung_task_sysctls[] = { > .data = &sysctl_hung_task_warnings, > .maxlen = sizeof(int), > .mode = 0644, >- .proc_handler = proc_dointvec_minmax, >+ .proc_handler = proc_dohung_task_warnings, > .extra1 = SYSCTL_NEG_ONE, > }, > { >-- Apart from that, LGTM. Reviewed-by: Lance Yang <lance.yang@linux.dev> Tested-by: Lance Yang <lance.yang@linux.dev> Nice and simple ;) Thanks for carrying this forward! Cheers, Lance ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 1/2] hung_task: Reset warning budget when problem gets resolved 2026-08-04 3:18 ` Lance Yang @ 2026-08-04 14:54 ` Aaron Tomlin 0 siblings, 0 replies; 8+ messages in thread From: Aaron Tomlin @ 2026-08-04 14:54 UTC (permalink / raw) To: Lance Yang Cc: akpm, mhiramat, pmladek, linux-kernel, david.laight.linux, neelx, sean, chjohnst, steve, mproche, nick.lange On Tue, Aug 04, 2026 at 11:18:39AM +0800, Lance Yang wrote: > > On Mon, Aug 03, 2026 at 08:15:41PM -0400, Aaron Tomlin wrote: > >From: Petr Mladek <pmladek@suse.com> > > > >sysctl_hung_task_warnings counts how many hung tasks are reported. > >The watchdog does not report anything once the limit is reached. > >Currently, this budget is decremented permanently, meaning the kernel is > >left blind to subsequent hung tasks even after the original issue resolves. > > > >Keep the global sysctl_hung_task_warnings intact, and instead decrement > >a copy (hung_task_warnings_printed) when warnings are printed. Reset > >the copy back to the configured sysctl_hung_task_warnings limit once the > >problem on the system gets resolved and check_hung_uninterruptible_tasks() > >detects no hung tasks in a check interval. > > > >Also keep the copy updated when the global sysctl_hung_task_warnings > >value is updated via sysctl, and update documentation to reflect > >the new behavior. > > > >Assisted-by: gemini-3.5-flash > >Signed-off-by: Petr Mladek <pmladek@suse.com> > >--- > > Documentation/admin-guide/sysctl/kernel.rst | 6 ++-- > > kernel/hung_task.c | 34 ++++++++++++++++----- > > 2 files changed, 31 insertions(+), 9 deletions(-) > > > >diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst > >index c6994e55d141..eaa1d837c739 100644 > >--- a/Documentation/admin-guide/sysctl/kernel.rst > >+++ b/Documentation/admin-guide/sysctl/kernel.rst > >@@ -459,8 +459,10 @@ hung_task_warnings > > ================== > > > > The maximum number of warnings to report. During a check interval > >-if a hung task is detected, this value is decreased by 1. > >-When this value reaches 0, no more warnings will be reported. > >+if a hung task is detected, the internal warning budget is decreased by 1. > >+When this budget reaches 0, no more warnings will be reported. > >+The warning budget is reset back to the configured limit when the problem on the > >+system gets resolved and no hung task is found. > > Maybe just go with this? > > " > The maximum number of warnings to report. During a check interval > if a hung task is detected, the internal warning budget is decreased by 1. > When this budget reaches 0, no more detailed warnings will be reported. > The warning budget is reset to the configured limit when no hung task is > found. > " Hi Lance, Acknowledged. > > > This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled. > > > > -1: report an infinite number of warnings. > >diff --git a/kernel/hung_task.c b/kernel/hung_task.c > >index 6fcc94ce4ca9..4e1fb0db79d1 100644 > >--- a/kernel/hung_task.c > >+++ b/kernel/hung_task.c > >@@ -59,6 +59,8 @@ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs; > > > > static int __read_mostly sysctl_hung_task_warnings = 10; > > > >+static int hung_task_warnings_printed = 10; > >+ > > static int __read_mostly did_panic; > > static bool hung_task_call_panic; > > > >@@ -247,9 +249,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, > > * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain > > * accordingly > > */ > >- if (sysctl_hung_task_warnings || hung_task_call_panic) { > >- if (sysctl_hung_task_warnings > 0) > >- sysctl_hung_task_warnings--; > >+ if (hung_task_warnings_printed || hung_task_call_panic) { > >+ if (hung_task_warnings_printed > 0) > >+ hung_task_warnings_printed--; > > pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n", > > t->comm, t->pid, t->in_iowait ? " in I/O wait" : "", > > (jiffies - t->last_switch_time) / HZ); > >@@ -264,7 +266,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, > > sched_show_task(t); > > debug_show_blocker(t, timeout); > > > >- if (!sysctl_hung_task_warnings) > >+ if (!hung_task_warnings_printed) > > pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n"); > > } > > > >@@ -304,7 +306,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) > > unsigned long last_break = jiffies; > > struct task_struct *g, *t; > > unsigned long this_round_count; > >- int need_warning = sysctl_hung_task_warnings; > >+ int need_warning = hung_task_warnings_printed; > > unsigned long si_mask = hung_task_si_mask; > > > > /* > >@@ -340,8 +342,10 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) > > unlock: > > rcu_read_unlock(); > > > >- if (!this_round_count) > >+ if (!this_round_count) { > > Hmm, should max_count, or more generally scan completion, be part of this > condition too? > > Maybe not, since a limited scan is already allowed to miss tasks ... > > If max_count stopped us early, we’d reset the budget even though a hung > task may still be later in the list ... > > I probably wouldn’t change it. Just wanted to throw the question out here. Thanks for raising the question! I agree with your intuition that no change is needed here. So, 'sysctl_hung_task_check_count' is designed to cap CPU usage during task iteration. If an admin caps max_count below the active thread count, requiring a complete scan before resetting would prevent the warning budget from ever resetting after system recovery. Furthermore, if a hung task happens to be beyond max_count in a given pass and gets missed, resetting the budget ensures that when khungtaskd does eventually reach and detect that hung task, it will log full diagnostic details rather than suppressing them. > >+ hung_task_warnings_printed = sysctl_hung_task_warnings; > > return; > >+ } > > > > if (need_warning || hung_task_call_panic) { > > si_mask |= SYS_INFO_LOCKS; > >@@ -425,6 +429,22 @@ static int proc_dohung_task_timeout_secs(const struct ctl_table *table, int writ > > return ret; > > } > > > >+static int proc_dohung_task_warnings(const struct ctl_table *table, int write, > >+ void *buffer, > >+ size_t *lenp, loff_t *ppos) > >+{ > >+ int ret; > >+ > >+ ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos); > >+ > >+ if (ret || !write) > >+ return ret; > >+ > >+ hung_task_warnings_printed = sysctl_hung_task_warnings; > >+ > >+ return 0; > >+} > >+ > > /* > > * This is needed for proc_doulongvec_minmax of sysctl_hung_task_timeout_secs > > * and hung_task_check_interval_secs > >@@ -480,7 +500,7 @@ static const struct ctl_table hung_task_sysctls[] = { > > .data = &sysctl_hung_task_warnings, > > .maxlen = sizeof(int), > > .mode = 0644, > >- .proc_handler = proc_dointvec_minmax, > >+ .proc_handler = proc_dohung_task_warnings, > > .extra1 = SYSCTL_NEG_ONE, > > }, > > { > >-- > > Apart from that, LGTM. > > Reviewed-by: Lance Yang <lance.yang@linux.dev> > Tested-by: Lance Yang <lance.yang@linux.dev> > > Nice and simple ;) Thanks for carrying this forward! > > Cheers, Lance Thank you! Kind regards, -- Aaron Tomlin ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 2/2] hung_task: Always print basic hung task info header 2026-08-04 0:15 [PATCH 0/2] hung_task: Improve warning budget handling and task reporting Aaron Tomlin 2026-08-04 0:15 ` [PATCH 1/2] hung_task: Reset warning budget when problem gets resolved Aaron Tomlin @ 2026-08-04 0:15 ` Aaron Tomlin 2026-08-04 3:19 ` Lance Yang 1 sibling, 1 reply; 8+ messages in thread From: Aaron Tomlin @ 2026-08-04 0:15 UTC (permalink / raw) To: akpm, lance.yang, mhiramat, pmladek Cc: linux-kernel, david.laight.linux, atomlin, neelx, sean, chjohnst, steve, mproche, nick.lange From: Petr Mladek <pmladek@suse.com> Currently, hung_task_info() completely suppresses any warning messages once the warning budget is reached. This leaves the system completely blind to any future hung tasks, even though we only wanted to suppress heavy process stack dumps. Modify hung_task_info() to always print the basic single-line error message about a hung task, regardless of whether the warning budget has been exhausted. Restrict only the verbose details (such as stack dumps, taint/release status, and blockers) to when the warning budget is intact or panic is triggered. Additionally, update the notice printed upon warning exhaustion to clarify that future reports will only omit process details rather than being completely suppressed. Assisted-by: gemini-3.5-flash Signed-off-by: Petr Mladek <pmladek@suse.com> --- kernel/hung_task.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/kernel/hung_task.c b/kernel/hung_task.c index 4e1fb0db79d1..6ebb3a87ac65 100644 --- a/kernel/hung_task.c +++ b/kernel/hung_task.c @@ -244,17 +244,19 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, hung_task_call_panic = true; } + /* Always print the blocked message */ + pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n", + t->comm, t->pid, t->in_iowait ? " in I/O wait" : "", + (jiffies - t->last_switch_time) / HZ); + /* * The given task did not get scheduled for more than * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain - * accordingly + * accordingly with full details if the budget is not exhausted. */ if (hung_task_warnings_printed || hung_task_call_panic) { if (hung_task_warnings_printed > 0) hung_task_warnings_printed--; - pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n", - t->comm, t->pid, t->in_iowait ? " in I/O wait" : "", - (jiffies - t->last_switch_time) / HZ); pr_err(" %s %s %.*s\n", print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, " "), @@ -267,7 +269,7 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout, debug_show_blocker(t, timeout); if (!hung_task_warnings_printed) - pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n"); + pr_info("Future hung task reports won't print details about each process, see sysctl kernel.hung_task_warnings\n"); } touch_nmi_watchdog(); -- 2.55.0 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] hung_task: Always print basic hung task info header 2026-08-04 0:15 ` [PATCH 2/2] hung_task: Always print basic hung task info header Aaron Tomlin @ 2026-08-04 3:19 ` Lance Yang 0 siblings, 0 replies; 8+ messages in thread From: Lance Yang @ 2026-08-04 3:19 UTC (permalink / raw) To: atomlin Cc: akpm, mhiramat, pmladek, linux-kernel, david.laight.linux, neelx, sean, chjohnst, steve, mproche, nick.lange, Lance Yang On Mon, Aug 03, 2026 at 08:15:42PM -0400, Aaron Tomlin wrote: >From: Petr Mladek <pmladek@suse.com> > >Currently, hung_task_info() completely suppresses any warning messages >once the warning budget is reached. This leaves the system completely >blind to any future hung tasks, even though we only wanted to suppress >heavy process stack dumps. > >Modify hung_task_info() to always print the basic single-line error message >about a hung task, regardless of whether the warning budget has been >exhausted. Restrict only the verbose details (such as stack dumps, >taint/release status, and blockers) to when the warning budget is intact >or panic is triggered. > >Additionally, update the notice printed upon warning exhaustion to clarify >that future reports will only omit process details rather than being completely >suppressed. > >Assisted-by: gemini-3.5-flash >Signed-off-by: Petr Mladek <pmladek@suse.com> >--- Reviewed-by: Lance Yang <lance.yang@linux.dev> ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2026-08-04 14:54 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-08-04 0:15 [PATCH 0/2] hung_task: Improve warning budget handling and task reporting Aaron Tomlin 2026-08-04 0:15 ` [PATCH 1/2] hung_task: Reset warning budget when problem gets resolved Aaron Tomlin 2026-08-04 2:24 ` Lance Yang 2026-08-04 14:19 ` Aaron Tomlin 2026-08-04 3:18 ` Lance Yang 2026-08-04 14:54 ` Aaron Tomlin 2026-08-04 0:15 ` [PATCH 2/2] hung_task: Always print basic hung task info header Aaron Tomlin 2026-08-04 3:19 ` Lance Yang
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox