* [PATCH v8 0/3] hung_task: Provide runtime reset interface for hung task detector
@ 2026-02-08 0:21 Aaron Tomlin
2026-02-08 0:21 ` [PATCH v8 1/3] hung_task: Refactor detection logic and atomicise detection count Aaron Tomlin
` (3 more replies)
0 siblings, 4 replies; 10+ messages in thread
From: Aaron Tomlin @ 2026-02-08 0:21 UTC (permalink / raw)
To: akpm, lance.yang, mhiramat, gregkh, pmladek, joel.granados
Cc: neelx, sean, mproche, chjohnst, nick.lange, linux-kernel
Hi Lance, Greg, Petr, Joel, Masami, Andrew,
This series introduces the ability to reset /proc/sys/kernel/hung_task_detect_count.
Writing a "0" value to this file atomically resets the counter of detected
hung tasks. This functionality provides system administrators with the
means to clear the cumulative diagnostic history following incident
resolution, thereby simplifying subsequent monitoring without necessitating
a system restart.
Please let me know your thoughts.
Changes since v7 [1]:
- Immediately update the global counter as soon as a hung task is
detected (Petr Mladek)
- Added Reviewed-by: tag from Masami Hiramatsu and Aaron Tomlin
Changes since v6 [2]:
- Decoupled the detection logic from the reporting logic.
check_hung_task() was renamed to hung_task_info() and the task_is_hung()
check was hoisted into the primary check_hung_uninterruptible_tasks()
loop (Petr Mladek)
- Changed the global statistic update mechanism from incremental atomic
increments (per task) to a single batched atomic_long_cmpxchg_relaxed()
at the end of the scan (Petr Mladek)
- Strengthened the memory ordering using atomic_long_read_acquire() and
atomic_long_cmpxchg_release(). This ensures strict synchronisation
between the scan and concurrent userspace resets (Petr Mladek)
- Updated the inline reporting comment to remove the hardcoded "2 minutes"
reference
Changes since v5 [3]:
- Introduced a preparatory patch (Joel Granados)
- Extended custom proc_handler to handle SYSCTL_USER_TO_KERN writes,
strictly validating that only a value of "0" is permitted for resets
(Joel Granados)
- Transitioned from atomic_long_inc_return_relaxed() to a more robust
read_acquire/cmpxchg_release pattern to ensure "All-or-Nothing" scan
updates (Petr Mladek)
- Re-introduce hung_task_diagnostics(). For better readability and
consistent metadata publication
Changes since v4 [4]:
- Added missing underflow check (Lance Yang)
Changes since v3 [5]:
- Use atomic operations to ensure cross-CPU visibility and prevent an integer underflow
- Use acquire/release semantics for memory ordering (Petr Mladek)
- Move quoted string to a single line (Petr Mladek)
- Remove variables coredump_msg and disable_msg to simplify code (Petr Mladek)
- Add trailing "\n" to all strings to ensure immediate console flushing (Petr Mladek)
- Improve the hung task counter documentation (Joel Granados)
- Reject non-zero writes with -EINVAL (Joel Granados)
- Translate to the new sysctl API (Petr Mladek)
Changes since v2 [6]:
- Avoided a needless double update to hung_task_detect_count (Lance Yang)
- Restored previous use of pr_err() for each message (Greg KH)
- Provided a complete descriptive comment for the helper
Changes since v1 [7]:
- Removed write-only sysfs attribute (Lance Yang)
- Modified procfs hung_task_detect_count instead (Lance Yang)
- Introduced a custom proc_handler
- Updated documentation (Lance Yang)
- Added 'static inline' as a hint to eliminate any function call overhead
- Removed clutter through encapsulation
[1]: https://lore.kernel.org/lkml/20260125135848.3356585-1-atomlin@atomlin.com/
[2]: https://lore.kernel.org/lkml/20260115023229.3028462-1-atomlin@atomlin.com/
[3]: https://lore.kernel.org/lkml/20251231004125.2380105-1-atomlin@atomlin.com/
[4]: https://lore.kernel.org/lkml/20251222014210.2032214-1-atomlin@atomlin.com/
[5]: https://lore.kernel.org/all/20251216030036.1822217-1-atomlin@atomlin.com/
[6]: https://lore.kernel.org/lkml/20251211033004.1628875-1-atomlin@atomlin.com/
[7]: https://lore.kernel.org/lkml/20251209041218.1583600-1-atomlin@atomlin.com/
Aaron Tomlin (2):
hung_task: Refactor detection logic and atomicise detection count
hung_task: Enable runtime reset of hung_task_detect_count
Petr Mladek (1):
hung_task: Increment the global counter immediately
Documentation/admin-guide/sysctl/kernel.rst | 3 +-
kernel/hung_task.c | 101 ++++++++++++++------
2 files changed, 75 insertions(+), 29 deletions(-)
--
2.51.0
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v8 1/3] hung_task: Refactor detection logic and atomicise detection count
2026-02-08 0:21 [PATCH v8 0/3] hung_task: Provide runtime reset interface for hung task detector Aaron Tomlin
@ 2026-02-08 0:21 ` Aaron Tomlin
2026-02-08 0:21 ` [PATCH v8 2/3] hung_task: Enable runtime reset of hung_task_detect_count Aaron Tomlin
` (2 subsequent siblings)
3 siblings, 0 replies; 10+ messages in thread
From: Aaron Tomlin @ 2026-02-08 0:21 UTC (permalink / raw)
To: akpm, lance.yang, mhiramat, gregkh, pmladek, joel.granados
Cc: neelx, sean, mproche, chjohnst, nick.lange, linux-kernel
The check_hung_task() function currently conflates two distinct
responsibilities: validating whether a task is hung and handling the
subsequent reporting (printing warnings, triggering panics, or
tracepoints).
This patch refactors the logic by introducing hung_task_info(), a
function dedicated solely to reporting. The actual detection check,
task_is_hung(), is hoisted into the primary loop within
check_hung_uninterruptible_tasks(). This separation clearly decouples
the mechanism of detection from the policy of reporting.
Furthermore, to facilitate future support for concurrent hung task
detection, the global sysctl_hung_task_detect_count variable is
converted from unsigned long to atomic_long_t. Consequently, the
counting logic is updated to accumulate the number of hung tasks locally
(this_round_count) during the iteration. The global counter is then
updated atomically via atomic_long_cmpxchg_relaxed() once the loop
concludes, rather than incrementally during the scan.
These changes are strictly preparatory and introduce no functional
change to the system's runtime behaviour.
Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
kernel/hung_task.c | 58 ++++++++++++++++++++++++++--------------------
1 file changed, 33 insertions(+), 25 deletions(-)
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index d2254c91450b..df10830ed9ef 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -36,7 +36,7 @@ static int __read_mostly sysctl_hung_task_check_count = PID_MAX_LIMIT;
/*
* Total number of tasks detected as hung since boot:
*/
-static unsigned long __read_mostly sysctl_hung_task_detect_count;
+static atomic_long_t sysctl_hung_task_detect_count = ATOMIC_LONG_INIT(0);
/*
* Limit number of tasks checked in a batch.
@@ -223,31 +223,29 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti
}
#endif
-static void check_hung_task(struct task_struct *t, unsigned long timeout,
- unsigned long prev_detect_count)
+/**
+ * hung_task_info - Print diagnostic details for a hung task
+ * @t: Pointer to the detected hung task.
+ * @timeout: Timeout threshold for detecting hung tasks
+ * @this_round_count: Count of hung tasks detected in the current iteration
+ *
+ * Print structured information about the specified hung task, if warnings
+ * are enabled or if the panic batch threshold is exceeded.
+ */
+static void hung_task_info(struct task_struct *t, unsigned long timeout,
+ unsigned long this_round_count)
{
- unsigned long total_hung_task;
-
- if (!task_is_hung(t, timeout))
- return;
-
- /*
- * This counter tracks the total number of tasks detected as hung
- * since boot.
- */
- sysctl_hung_task_detect_count++;
-
- total_hung_task = sysctl_hung_task_detect_count - prev_detect_count;
trace_sched_process_hang(t);
- if (sysctl_hung_task_panic && total_hung_task >= sysctl_hung_task_panic) {
+ if (sysctl_hung_task_panic && this_round_count >= sysctl_hung_task_panic) {
console_verbose();
hung_task_call_panic = true;
}
/*
- * Ok, the task did not get scheduled for more than 2 minutes,
- * complain:
+ * The given task did not get scheduled for more than
+ * CONFIG_DEFAULT_HUNG_TASK_TIMEOUT. Therefore, complain
+ * accordingly
*/
if (sysctl_hung_task_warnings || hung_task_call_panic) {
if (sysctl_hung_task_warnings > 0)
@@ -297,18 +295,18 @@ static bool rcu_lock_break(struct task_struct *g, struct task_struct *t)
/*
* Check whether a TASK_UNINTERRUPTIBLE does not get woken up for
- * a really long time (120 seconds). If that happens, print out
- * a warning.
+ * a really long time. If that happens, print out a warning.
*/
static void check_hung_uninterruptible_tasks(unsigned long timeout)
{
int max_count = sysctl_hung_task_check_count;
unsigned long last_break = jiffies;
struct task_struct *g, *t;
- unsigned long prev_detect_count = sysctl_hung_task_detect_count;
+ unsigned long total_count, this_round_count;
int need_warning = sysctl_hung_task_warnings;
unsigned long si_mask = hung_task_si_mask;
+ total_count = atomic_long_read(&sysctl_hung_task_detect_count);
/*
* If the system crashed already then all bets are off,
* do not report extra hung tasks:
@@ -316,10 +314,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
if (test_taint(TAINT_DIE) || did_panic)
return;
-
+ this_round_count = 0;
rcu_read_lock();
for_each_process_thread(g, t) {
-
if (!max_count--)
goto unlock;
if (time_after(jiffies, last_break + HUNG_TASK_LOCK_BREAK)) {
@@ -328,14 +325,25 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
last_break = jiffies;
}
- check_hung_task(t, timeout, prev_detect_count);
+ if (task_is_hung(t, timeout)) {
+ this_round_count++;
+ hung_task_info(t, timeout, this_round_count);
+ }
}
unlock:
rcu_read_unlock();
- if (!(sysctl_hung_task_detect_count - prev_detect_count))
+ if (!this_round_count)
return;
+ /*
+ * This counter tracks the total number of tasks detected as hung
+ * since boot.
+ */
+ atomic_long_cmpxchg_relaxed(&sysctl_hung_task_detect_count,
+ total_count, total_count +
+ this_round_count);
+
if (need_warning || hung_task_call_panic) {
si_mask |= SYS_INFO_LOCKS;
--
2.51.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v8 2/3] hung_task: Enable runtime reset of hung_task_detect_count
2026-02-08 0:21 [PATCH v8 0/3] hung_task: Provide runtime reset interface for hung task detector Aaron Tomlin
2026-02-08 0:21 ` [PATCH v8 1/3] hung_task: Refactor detection logic and atomicise detection count Aaron Tomlin
@ 2026-02-08 0:21 ` Aaron Tomlin
2026-02-20 7:06 ` Joel Granados
2026-02-08 0:21 ` [PATCH v8 3/3] hung_task: Increment the global counter immediately Aaron Tomlin
2026-02-08 4:11 ` [PATCH v8 0/3] hung_task: Provide runtime reset interface for hung task detector Lance Yang
3 siblings, 1 reply; 10+ messages in thread
From: Aaron Tomlin @ 2026-02-08 0:21 UTC (permalink / raw)
To: akpm, lance.yang, mhiramat, gregkh, pmladek, joel.granados
Cc: neelx, sean, mproche, chjohnst, nick.lange, linux-kernel
Currently, the hung_task_detect_count sysctl provides a cumulative count
of hung tasks since boot. In long-running, high-availability
environments, this counter may lose its utility if it cannot be reset
once an incident has been resolved. Furthermore, the previous
implementation relied upon implicit ordering, which could not strictly
guarantee that diagnostic metadata published by one CPU was visible to
the panic logic on another.
This patch introduces the capability to reset the detection count by
writing "0" to the hung_task_detect_count sysctl. The proc_handler logic
has been updated to validate this input and atomically reset the
counter.
The synchronisation of sysctl_hung_task_detect_count relies upon a
transactional model to ensure the integrity of the detection counter
against concurrent resets from userspace. The application of
atomic_long_read_acquire() and atomic_long_cmpxchg_release() is correct
and provides the following guarantees:
1. Prevention of Load-Store Reordering via Acquire Semantics By
utilising atomic_long_read_acquire() to snapshot the counter
before initiating the task traversal, we establish a strict
memory barrier. This prevents the compiler or hardware from
reordering the initial load to a point later in the scan. Without
this "acquire" barrier, a delayed load could potentially read a
"0" value resulting from a userspace reset that occurred
mid-scan. This would lead to the subsequent cmpxchg succeeding
erroneously, thereby overwriting the user's reset with stale
increment data.
2. Atomicity of the "Commit" Phase via Release Semantics The
atomic_long_cmpxchg_release() serves as the transaction's commit
point. The "release" barrier ensures that all diagnostic
recordings and task-state observations made during the scan are
globally visible before the counter is incremented.
3. Race Condition Resolution This pairing effectively detects any
"out-of-band" reset of the counter. If
sysctl_hung_task_detect_count is modified via the procfs
interface during the scan, the final cmpxchg will detect the
discrepancy between the current value and the "acquire" snapshot.
Consequently, the update will fail, ensuring that a reset command
from the administrator is prioritised over a scan that may have
been invalidated by that very reset.
Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
Documentation/admin-guide/sysctl/kernel.rst | 3 +-
kernel/hung_task.c | 58 ++++++++++++++++++---
2 files changed, 53 insertions(+), 8 deletions(-)
diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index 239da22c4e28..68da4235225a 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -418,7 +418,8 @@ hung_task_detect_count
======================
Indicates the total number of tasks that have been detected as hung since
-the system boot.
+the system boot or since the counter was reset. The counter is zeroed when
+a value of 0 is written.
This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index df10830ed9ef..350093de0535 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -306,7 +306,11 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
int need_warning = sysctl_hung_task_warnings;
unsigned long si_mask = hung_task_si_mask;
- total_count = atomic_long_read(&sysctl_hung_task_detect_count);
+ /*
+ * The counter might get reset. Remember the initial value.
+ * Acquire prevents reordering task checks before this point.
+ */
+ total_count = atomic_long_read_acquire(&sysctl_hung_task_detect_count);
/*
* If the system crashed already then all bets are off,
* do not report extra hung tasks:
@@ -337,10 +341,11 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
return;
/*
- * This counter tracks the total number of tasks detected as hung
- * since boot.
+ * Do not count this round when the global counter has been reset
+ * during this check. Release ensures we see all hang details
+ * recorded during the scan.
*/
- atomic_long_cmpxchg_relaxed(&sysctl_hung_task_detect_count,
+ atomic_long_cmpxchg_release(&sysctl_hung_task_detect_count,
total_count, total_count +
this_round_count);
@@ -366,6 +371,46 @@ static long hung_timeout_jiffies(unsigned long last_checked,
}
#ifdef CONFIG_SYSCTL
+
+/**
+ * proc_dohung_task_detect_count - proc handler for hung_task_detect_count
+ * @table: Pointer to the struct ctl_table definition for this proc entry
+ * @dir: Flag indicating the operation
+ * @buffer: User space buffer for data transfer
+ * @lenp: Pointer to the length of the data being transferred
+ * @ppos: Pointer to the current file offset
+ *
+ * This handler is used for reading the current hung task detection count
+ * and for resetting it to zero when a write operation is performed using a
+ * zero value only.
+ * Return: 0 on success, or a negative error code on failure.
+ */
+static int proc_dohung_task_detect_count(const struct ctl_table *table, int dir,
+ void *buffer, size_t *lenp, loff_t *ppos)
+{
+ unsigned long detect_count;
+ struct ctl_table proxy_table;
+ int err;
+
+ proxy_table = *table;
+ proxy_table.data = &detect_count;
+
+ if (SYSCTL_KERN_TO_USER(dir))
+ detect_count = atomic_long_read(&sysctl_hung_task_detect_count);
+
+ err = proc_doulongvec_minmax(&proxy_table, dir, buffer, lenp, ppos);
+ if (err < 0)
+ return err;
+
+ if (SYSCTL_USER_TO_KERN(dir)) {
+ if (detect_count)
+ return -EINVAL;
+ atomic_long_set(&sysctl_hung_task_detect_count, 0);
+ }
+
+ return 0;
+}
+
/*
* Process updating of timeout sysctl
*/
@@ -446,10 +491,9 @@ static const struct ctl_table hung_task_sysctls[] = {
},
{
.procname = "hung_task_detect_count",
- .data = &sysctl_hung_task_detect_count,
.maxlen = sizeof(unsigned long),
- .mode = 0444,
- .proc_handler = proc_doulongvec_minmax,
+ .mode = 0644,
+ .proc_handler = proc_dohung_task_detect_count,
},
{
.procname = "hung_task_sys_info",
--
2.51.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v8 3/3] hung_task: Increment the global counter immediately
2026-02-08 0:21 [PATCH v8 0/3] hung_task: Provide runtime reset interface for hung task detector Aaron Tomlin
2026-02-08 0:21 ` [PATCH v8 1/3] hung_task: Refactor detection logic and atomicise detection count Aaron Tomlin
2026-02-08 0:21 ` [PATCH v8 2/3] hung_task: Enable runtime reset of hung_task_detect_count Aaron Tomlin
@ 2026-02-08 0:21 ` Aaron Tomlin
2026-02-08 4:13 ` Lance Yang
2026-02-08 4:11 ` [PATCH v8 0/3] hung_task: Provide runtime reset interface for hung task detector Lance Yang
3 siblings, 1 reply; 10+ messages in thread
From: Aaron Tomlin @ 2026-02-08 0:21 UTC (permalink / raw)
To: akpm, lance.yang, mhiramat, gregkh, pmladek, joel.granados
Cc: neelx, sean, mproche, chjohnst, nick.lange, linux-kernel
From: Petr Mladek <pmladek@suse.com>
A recent change allowed to reset the global counter of hung tasks using
the sysctl interface. A potential race with the regular check has been
solved by updating the global counter only once at the end of the check.
However, the hung task check can take a significant amount of time,
particularly when task information is being dumped to slow serial
consoles. Some users monitor this global counter to trigger immediate
migration of critical containers. Delaying the increment until the
full check completes postpones these high-priority rescue operations.
Update the global counter as soon as a hung task is detected. Since
the value is read asynchronously, a relaxed atomic operation is
sufficient.
Reported-by: Lance Yang <lance.yang@linux.dev>
Closes: https://lore.kernel.org/r/f239e00f-4282-408d-b172-0f9885f4b01b@linux.dev
Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Petr Mladek <pmladek@suse.com>
---
kernel/hung_task.c | 23 ++++++++---------------
1 file changed, 8 insertions(+), 15 deletions(-)
diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 350093de0535..8bc043fbe89c 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -302,15 +302,10 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
int max_count = sysctl_hung_task_check_count;
unsigned long last_break = jiffies;
struct task_struct *g, *t;
- unsigned long total_count, this_round_count;
+ unsigned long this_round_count;
int need_warning = sysctl_hung_task_warnings;
unsigned long si_mask = hung_task_si_mask;
- /*
- * The counter might get reset. Remember the initial value.
- * Acquire prevents reordering task checks before this point.
- */
- total_count = atomic_long_read_acquire(&sysctl_hung_task_detect_count);
/*
* If the system crashed already then all bets are off,
* do not report extra hung tasks:
@@ -330,6 +325,13 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
}
if (task_is_hung(t, timeout)) {
+ /*
+ * Increment the global counter so that userspace could
+ * start migrating tasks ASAP. But count the current
+ * round separately because userspace could reset
+ * the global counter at any time.
+ */
+ atomic_long_inc(&sysctl_hung_task_detect_count);
this_round_count++;
hung_task_info(t, timeout, this_round_count);
}
@@ -340,15 +342,6 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
if (!this_round_count)
return;
- /*
- * Do not count this round when the global counter has been reset
- * during this check. Release ensures we see all hang details
- * recorded during the scan.
- */
- atomic_long_cmpxchg_release(&sysctl_hung_task_detect_count,
- total_count, total_count +
- this_round_count);
-
if (need_warning || hung_task_call_panic) {
si_mask |= SYS_INFO_LOCKS;
--
2.51.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH v8 0/3] hung_task: Provide runtime reset interface for hung task detector
2026-02-08 0:21 [PATCH v8 0/3] hung_task: Provide runtime reset interface for hung task detector Aaron Tomlin
` (2 preceding siblings ...)
2026-02-08 0:21 ` [PATCH v8 3/3] hung_task: Increment the global counter immediately Aaron Tomlin
@ 2026-02-08 4:11 ` Lance Yang
2026-02-08 4:19 ` Lance Yang
2026-02-08 19:49 ` Aaron Tomlin
3 siblings, 2 replies; 10+ messages in thread
From: Lance Yang @ 2026-02-08 4:11 UTC (permalink / raw)
To: Aaron Tomlin
Cc: neelx, sean, akpm, mproche, chjohnst, nick.lange, gregkh,
linux-kernel, joel.granados, pmladek, mhiramat
On 2026/2/8 08:21, Aaron Tomlin wrote:
> Hi Lance, Greg, Petr, Joel, Masami, Andrew,
>
> This series introduces the ability to reset /proc/sys/kernel/hung_task_detect_count.
>
> Writing a "0" value to this file atomically resets the counter of detected
> hung tasks. This functionality provides system administrators with the
> means to clear the cumulative diagnostic history following incident
> resolution, thereby simplifying subsequent monitoring without necessitating
> a system restart.
>
> Please let me know your thoughts.
>
>
> Changes since v7 [1]:
> - Immediately update the global counter as soon as a hung task is
> detected (Petr Mladek)
> - Added Reviewed-by: tag from Masami Hiramatsu and Aaron Tomlin
Looks like you forgot to include Petr's RB on #01 and #02 ... IIRC
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v8 3/3] hung_task: Increment the global counter immediately
2026-02-08 0:21 ` [PATCH v8 3/3] hung_task: Increment the global counter immediately Aaron Tomlin
@ 2026-02-08 4:13 ` Lance Yang
0 siblings, 0 replies; 10+ messages in thread
From: Lance Yang @ 2026-02-08 4:13 UTC (permalink / raw)
To: Aaron Tomlin, pmladek
Cc: neelx, sean, mproche, chjohnst, nick.lange, linux-kernel, gregkh,
mhiramat, akpm, joel.granados
On 2026/2/8 08:21, Aaron Tomlin wrote:
> From: Petr Mladek <pmladek@suse.com>
>
> A recent change allowed to reset the global counter of hung tasks using
> the sysctl interface. A potential race with the regular check has been
> solved by updating the global counter only once at the end of the check.
>
> However, the hung task check can take a significant amount of time,
> particularly when task information is being dumped to slow serial
> consoles. Some users monitor this global counter to trigger immediate
> migration of critical containers. Delaying the increment until the
> full check completes postpones these high-priority rescue operations.
>
> Update the global counter as soon as a hung task is detected. Since
> the value is read asynchronously, a relaxed atomic operation is
> sufficient.
>
> Reported-by: Lance Yang <lance.yang@linux.dev>
> Closes: https://lore.kernel.org/r/f239e00f-4282-408d-b172-0f9885f4b01b@linux.dev
> Reviewed-by: Aaron Tomlin <atomlin@atomlin.com>
> Signed-off-by: Petr Mladek <pmladek@suse.com>
> ---
Looks good to me, so:
Reviewed-by: Lance Yang <lance.yang@linux.dev>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v8 0/3] hung_task: Provide runtime reset interface for hung task detector
2026-02-08 4:11 ` [PATCH v8 0/3] hung_task: Provide runtime reset interface for hung task detector Lance Yang
@ 2026-02-08 4:19 ` Lance Yang
2026-02-08 19:49 ` Aaron Tomlin
1 sibling, 0 replies; 10+ messages in thread
From: Lance Yang @ 2026-02-08 4:19 UTC (permalink / raw)
To: Aaron Tomlin
Cc: neelx, sean, akpm, mproche, chjohnst, nick.lange, gregkh,
linux-kernel, joel.granados, pmladek, mhiramat
On 2026/2/8 12:11, Lance Yang wrote:
>
>
> On 2026/2/8 08:21, Aaron Tomlin wrote:
>> Hi Lance, Greg, Petr, Joel, Masami, Andrew,
>>
>> This series introduces the ability to reset /proc/sys/kernel/
>> hung_task_detect_count.
>>
>> Writing a "0" value to this file atomically resets the counter of
>> detected
>> hung tasks. This functionality provides system administrators with the
>> means to clear the cumulative diagnostic history following incident
>> resolution, thereby simplifying subsequent monitoring without
>> necessitating
>> a system restart.
>>
>> Please let me know your thoughts.
>>
>>
>> Changes since v7 [1]:
>> - Immediately update the global counter as soon as a hung task is
>> detected (Petr Mladek)
>> - Added Reviewed-by: tag from Masami Hiramatsu and Aaron Tomlin
>
> Looks like you forgot to include Petr's RB on #01 and #02 ... IIRC
But no need to resend this series - Andrew could add them when applying ;p
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v8 0/3] hung_task: Provide runtime reset interface for hung task detector
2026-02-08 4:11 ` [PATCH v8 0/3] hung_task: Provide runtime reset interface for hung task detector Lance Yang
2026-02-08 4:19 ` Lance Yang
@ 2026-02-08 19:49 ` Aaron Tomlin
1 sibling, 0 replies; 10+ messages in thread
From: Aaron Tomlin @ 2026-02-08 19:49 UTC (permalink / raw)
To: Lance Yang
Cc: neelx, sean, akpm, mproche, chjohnst, nick.lange, gregkh,
linux-kernel, joel.granados, pmladek, mhiramat
[-- Attachment #1: Type: text/plain, Size: 666 bytes --]
On Sun, Feb 08, 2026 at 12:11:47PM +0800, Lance Yang wrote:
> > Changes since v7 [1]:
> > - Immediately update the global counter as soon as a hung task is
> > detected (Petr Mladek)
> > - Added Reviewed-by: tag from Masami Hiramatsu and Aaron Tomlin
>
> Looks like you forgot to include Petr's RB on #01 and #02 ... IIRC
Hi Lance,
You are quite right [1][2]. Many thanks for pointing this out.
Andrew,
Can you please manually add: Reviewed-By: Petr Mladek <pmladek@suse.com>
[1]: https://lore.kernel.org/lkml/aYCfpS--vgVMzYWU@pathway/
[2]: https://lore.kernel.org/lkml/aYCl9iTmr175xvwN@pathway/
Kind regards,
--
Aaron Tomlin
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v8 2/3] hung_task: Enable runtime reset of hung_task_detect_count
2026-02-08 0:21 ` [PATCH v8 2/3] hung_task: Enable runtime reset of hung_task_detect_count Aaron Tomlin
@ 2026-02-20 7:06 ` Joel Granados
2026-03-02 22:30 ` Aaron Tomlin
0 siblings, 1 reply; 10+ messages in thread
From: Joel Granados @ 2026-02-20 7:06 UTC (permalink / raw)
To: Aaron Tomlin
Cc: akpm, lance.yang, mhiramat, gregkh, pmladek, neelx, sean, mproche,
chjohnst, nick.lange, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 7016 bytes --]
On Sat, Feb 07, 2026 at 07:21:15PM -0500, Aaron Tomlin wrote:
> Currently, the hung_task_detect_count sysctl provides a cumulative count
> of hung tasks since boot. In long-running, high-availability
> environments, this counter may lose its utility if it cannot be reset
> once an incident has been resolved. Furthermore, the previous
> implementation relied upon implicit ordering, which could not strictly
> guarantee that diagnostic metadata published by one CPU was visible to
> the panic logic on another.
>
> This patch introduces the capability to reset the detection count by
> writing "0" to the hung_task_detect_count sysctl. The proc_handler logic
> has been updated to validate this input and atomically reset the
> counter.
>
> The synchronisation of sysctl_hung_task_detect_count relies upon a
> transactional model to ensure the integrity of the detection counter
> against concurrent resets from userspace. The application of
> atomic_long_read_acquire() and atomic_long_cmpxchg_release() is correct
> and provides the following guarantees:
>
> 1. Prevention of Load-Store Reordering via Acquire Semantics By
> utilising atomic_long_read_acquire() to snapshot the counter
> before initiating the task traversal, we establish a strict
> memory barrier. This prevents the compiler or hardware from
> reordering the initial load to a point later in the scan. Without
> this "acquire" barrier, a delayed load could potentially read a
> "0" value resulting from a userspace reset that occurred
> mid-scan. This would lead to the subsequent cmpxchg succeeding
> erroneously, thereby overwriting the user's reset with stale
> increment data.
>
> 2. Atomicity of the "Commit" Phase via Release Semantics The
> atomic_long_cmpxchg_release() serves as the transaction's commit
> point. The "release" barrier ensures that all diagnostic
> recordings and task-state observations made during the scan are
> globally visible before the counter is incremented.
>
> 3. Race Condition Resolution This pairing effectively detects any
> "out-of-band" reset of the counter. If
> sysctl_hung_task_detect_count is modified via the procfs
> interface during the scan, the final cmpxchg will detect the
> discrepancy between the current value and the "acquire" snapshot.
> Consequently, the update will fail, ensuring that a reset command
> from the administrator is prioritised over a scan that may have
> been invalidated by that very reset.
>
> Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
> Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
> ---
> Documentation/admin-guide/sysctl/kernel.rst | 3 +-
> kernel/hung_task.c | 58 ++++++++++++++++++---
> 2 files changed, 53 insertions(+), 8 deletions(-)
>
> diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
> index 239da22c4e28..68da4235225a 100644
> --- a/Documentation/admin-guide/sysctl/kernel.rst
> +++ b/Documentation/admin-guide/sysctl/kernel.rst
> @@ -418,7 +418,8 @@ hung_task_detect_count
> ======================
>
> Indicates the total number of tasks that have been detected as hung since
> -the system boot.
> +the system boot or since the counter was reset. The counter is zeroed when
> +a value of 0 is written.
>
> This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled.
>
> diff --git a/kernel/hung_task.c b/kernel/hung_task.c
> index df10830ed9ef..350093de0535 100644
> --- a/kernel/hung_task.c
> +++ b/kernel/hung_task.c
> @@ -306,7 +306,11 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
> int need_warning = sysctl_hung_task_warnings;
> unsigned long si_mask = hung_task_si_mask;
>
> - total_count = atomic_long_read(&sysctl_hung_task_detect_count);
> + /*
> + * The counter might get reset. Remember the initial value.
> + * Acquire prevents reordering task checks before this point.
> + */
> + total_count = atomic_long_read_acquire(&sysctl_hung_task_detect_count);
> /*
> * If the system crashed already then all bets are off,
> * do not report extra hung tasks:
> @@ -337,10 +341,11 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)
> return;
>
> /*
> - * This counter tracks the total number of tasks detected as hung
> - * since boot.
> + * Do not count this round when the global counter has been reset
> + * during this check. Release ensures we see all hang details
> + * recorded during the scan.
> */
> - atomic_long_cmpxchg_relaxed(&sysctl_hung_task_detect_count,
> + atomic_long_cmpxchg_release(&sysctl_hung_task_detect_count,
> total_count, total_count +
> this_round_count);
>
> @@ -366,6 +371,46 @@ static long hung_timeout_jiffies(unsigned long last_checked,
> }
>
> #ifdef CONFIG_SYSCTL
> +
> +/**
> + * proc_dohung_task_detect_count - proc handler for hung_task_detect_count
> + * @table: Pointer to the struct ctl_table definition for this proc entry
> + * @dir: Flag indicating the operation
> + * @buffer: User space buffer for data transfer
> + * @lenp: Pointer to the length of the data being transferred
> + * @ppos: Pointer to the current file offset
> + *
> + * This handler is used for reading the current hung task detection count
> + * and for resetting it to zero when a write operation is performed using a
> + * zero value only.
> + * Return: 0 on success, or a negative error code on failure.
> + */
> +static int proc_dohung_task_detect_count(const struct ctl_table *table, int dir,
> + void *buffer, size_t *lenp, loff_t *ppos)
> +{
> + unsigned long detect_count;
> + struct ctl_table proxy_table;
> + int err;
> +
> + proxy_table = *table;
> + proxy_table.data = &detect_count;
> +
> + if (SYSCTL_KERN_TO_USER(dir))
> + detect_count = atomic_long_read(&sysctl_hung_task_detect_count);
> +
> + err = proc_doulongvec_minmax(&proxy_table, dir, buffer, lenp, ppos);
> + if (err < 0)
> + return err;
> +
> + if (SYSCTL_USER_TO_KERN(dir)) {
> + if (detect_count)
> + return -EINVAL;
> + atomic_long_set(&sysctl_hung_task_detect_count, 0);
> + }
> +
> + return 0;
> +}
> +
> /*
> * Process updating of timeout sysctl
> */
> @@ -446,10 +491,9 @@ static const struct ctl_table hung_task_sysctls[] = {
> },
> {
> .procname = "hung_task_detect_count",
> - .data = &sysctl_hung_task_detect_count,
> .maxlen = sizeof(unsigned long),
> - .mode = 0444,
> - .proc_handler = proc_doulongvec_minmax,
> + .mode = 0644,
> + .proc_handler = proc_dohung_task_detect_count,
> },
> {
> .procname = "hung_task_sys_info",
> --
> 2.51.0
>
Feel free to add (if it is not too late)
Reviewed-by: Joel Granados <joel.granados@kernel.org>
--
Joel Granados
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v8 2/3] hung_task: Enable runtime reset of hung_task_detect_count
2026-02-20 7:06 ` Joel Granados
@ 2026-03-02 22:30 ` Aaron Tomlin
0 siblings, 0 replies; 10+ messages in thread
From: Aaron Tomlin @ 2026-03-02 22:30 UTC (permalink / raw)
To: Joel Granados, akpm
Cc: lance.yang, mhiramat, gregkh, pmladek, neelx, sean, mproche,
chjohnst, nick.lange, linux-kernel
On Fri, Feb 20, 2026 at 08:06:58AM +0100, Joel Granados wrote:
> Feel free to add (if it is not too late)
> Reviewed-by: Joel Granados <joel.granados@kernel.org>
Hi Joel Granados,
Thank you. I don't believe so.
Hi Andrew,
Shall I prepare a v9 to include all "Reviewed-by:" tags?
Currently as follows:
1. [PATCH v8 1/3] hung_task: Refactor detection logic and atomicise detection count
Reviewed-By: Petr Mladek <pmladek@suse.com>
2. [PATCH v8 2/3] hung_task: Enable runtime reset of hung_task_detect_count
Reviewed-By: Petr Mladek <pmladek@suse.com>
Reviewed-by: Joel Granados <joel.granados@kernel.org>
3. [PATCH v8 3/3] hung_task: Increment the global counter immediately
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Kind regards,
--
Aaron Tomlin
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2026-03-02 22:30 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-02-08 0:21 [PATCH v8 0/3] hung_task: Provide runtime reset interface for hung task detector Aaron Tomlin
2026-02-08 0:21 ` [PATCH v8 1/3] hung_task: Refactor detection logic and atomicise detection count Aaron Tomlin
2026-02-08 0:21 ` [PATCH v8 2/3] hung_task: Enable runtime reset of hung_task_detect_count Aaron Tomlin
2026-02-20 7:06 ` Joel Granados
2026-03-02 22:30 ` Aaron Tomlin
2026-02-08 0:21 ` [PATCH v8 3/3] hung_task: Increment the global counter immediately Aaron Tomlin
2026-02-08 4:13 ` Lance Yang
2026-02-08 4:11 ` [PATCH v8 0/3] hung_task: Provide runtime reset interface for hung task detector Lance Yang
2026-02-08 4:19 ` Lance Yang
2026-02-08 19:49 ` Aaron Tomlin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox