[v2 PATCH 0/1] hung_task: Explicitly report I/O wait state in log output

[v2 PATCH 0/1] hung_task: Explicitly report I/O wait state in log output

[Aaron Tomlin]

Hi Lance, Masami, Petr, Andrew,

Currently, the hung task reporting mechanism indiscriminately labels all
TASK_UNINTERRUPTIBLE (D) tasks as "blocked", irrespective of whether they
are awaiting I/O completion or kernel locking primitives. This ambiguity
compels system administrators to manually inspect stack traces to discern
whether the delay stems from an I/O wait (typically indicative of
hardware or filesystem anomalies) or software contention. Such detailed
analysis is not always immediately accessible to system administrators
or support engineers.

To address this, this patch utilises the existing in_iowait field within
struct task_struct to augment the failure report. If the task is blocked
due to I/O (e.g., via io_schedule_prepare()), the log message is updated
to explicitly state "blocked in I/O wait".

Examples:
        - Standard Block: "INFO: task bash:123 blocked for more than 120
          seconds".

        - I/O Block: "INFO: task dd:456 blocked in I/O wait for more than
          120 seconds".

Accessing in_iowait is safe in this context. The detector holds
rcu_read_lock() within check_hung_uninterruptible_tasks(), ensuring the
task structure remains valid in memory. Furthermore, as the task is
confirmed to be in a persistent TASK_UNINTERRUPTIBLE state, it cannot
modify its own in_iowait flag, rendering the read operation stable and
free from data races.

Please let me know your thoughts.


Changes since v1 [1]:
 - Updated patch subject to reflect the new output format (was: "hung_task:
   Differentiate between I/O and Lock/Resource waits")
 - Changed log output from "D (Disk I/O)" to "blocked in I/O wait"
   (Masami Hiramatsu)

[1]: https://lore.kernel.org/lkml/20260125203905.3393869-1-atomlin@atomlin.com/

Aaron Tomlin (1):
  hung_task: Explicitly report I/O wait state in log output

 kernel/hung_task.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

-- 
2.51.0

[v2 PATCH 1/1] hung_task: Explicitly report I/O wait state in log output

[Aaron Tomlin]

Currently, the hung task reporting mechanism indiscriminately labels all
TASK_UNINTERRUPTIBLE (D) tasks as "blocked", irrespective of whether they
are awaiting I/O completion or kernel locking primitives. This ambiguity
compels system administrators to manually inspect stack traces to discern
whether the delay stems from an I/O wait (typically indicative of
hardware or filesystem anomalies) or software contention. Such detailed
analysis is not always immediately accessible to system administrators
or support engineers.

To address this, this patch utilises the existing in_iowait field within
struct task_struct to augment the failure report. If the task is blocked
due to I/O (e.g., via io_schedule_prepare()), the log message is updated
to explicitly state "blocked in I/O wait".

Examples:
        - Standard Block: "INFO: task bash:123 blocked for more than 120
          seconds".

        - I/O Block: "INFO: task dd:456 blocked in I/O wait for more than
          120 seconds".

Accessing in_iowait is safe in this context. The detector holds
rcu_read_lock() within check_hung_uninterruptible_tasks(), ensuring the
task structure remains valid in memory. Furthermore, as the task is
confirmed to be in a persistent TASK_UNINTERRUPTIBLE state, it cannot
modify its own in_iowait flag, rendering the read operation stable and
free from data races.

Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
 kernel/hung_task.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/hung_task.c b/kernel/hung_task.c
index 350093de0535..abfbb5d9eeee 100644
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -250,8 +250,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 	if (sysctl_hung_task_warnings || hung_task_call_panic) {
 		if (sysctl_hung_task_warnings > 0)
 			sysctl_hung_task_warnings--;
-		pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n",
-		       t->comm, t->pid, (jiffies - t->last_switch_time) / HZ);
+		pr_err("INFO: task %s:%d blocked %s for more than %ld seconds.\n",
+		       t->comm, t->pid, t->in_iowait ? "in I/O wait" : "",
+		       (jiffies - t->last_switch_time) / HZ);
 		pr_err("      %s %s %.*s\n",
 			print_tainted(), init_utsname()->release,
 			(int)strcspn(init_utsname()->version, " "),
-- 
2.51.0

Re: [v2 PATCH 1/1] hung_task: Explicitly report I/O wait state in log output

[Masami Hiramatsu]

On Wed, 28 Jan 2026 15:45:16 -0500
Aaron Tomlin <atomlin@atomlin.com> wrote:

> Currently, the hung task reporting mechanism indiscriminately labels all
> TASK_UNINTERRUPTIBLE (D) tasks as "blocked", irrespective of whether they
> are awaiting I/O completion or kernel locking primitives. This ambiguity
> compels system administrators to manually inspect stack traces to discern
> whether the delay stems from an I/O wait (typically indicative of
> hardware or filesystem anomalies) or software contention. Such detailed
> analysis is not always immediately accessible to system administrators
> or support engineers.
> 
> To address this, this patch utilises the existing in_iowait field within
> struct task_struct to augment the failure report. If the task is blocked
> due to I/O (e.g., via io_schedule_prepare()), the log message is updated
> to explicitly state "blocked in I/O wait".
> 
> Examples:
>         - Standard Block: "INFO: task bash:123 blocked for more than 120
>           seconds".
> 
>         - I/O Block: "INFO: task dd:456 blocked in I/O wait for more than
>           120 seconds".
> 
> Accessing in_iowait is safe in this context. The detector holds
> rcu_read_lock() within check_hung_uninterruptible_tasks(), ensuring the
> task structure remains valid in memory. Furthermore, as the task is
> confirmed to be in a persistent TASK_UNINTERRUPTIBLE state, it cannot
> modify its own in_iowait flag, rendering the read operation stable and
> free from data races.
> 
> Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>

Looks good to me.

Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>

Thanks,


> ---
>  kernel/hung_task.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/hung_task.c b/kernel/hung_task.c
> index 350093de0535..abfbb5d9eeee 100644
> --- a/kernel/hung_task.c
> +++ b/kernel/hung_task.c
> @@ -250,8 +250,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
>  	if (sysctl_hung_task_warnings || hung_task_call_panic) {
>  		if (sysctl_hung_task_warnings > 0)
>  			sysctl_hung_task_warnings--;
> -		pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n",
> -		       t->comm, t->pid, (jiffies - t->last_switch_time) / HZ);
> +		pr_err("INFO: task %s:%d blocked %s for more than %ld seconds.\n",
> +		       t->comm, t->pid, t->in_iowait ? "in I/O wait" : "",
> +		       (jiffies - t->last_switch_time) / HZ);
>  		pr_err("      %s %s %.*s\n",
>  			print_tainted(), init_utsname()->release,
>  			(int)strcspn(init_utsname()->version, " "),
> -- 
> 2.51.0
> 


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [v2 PATCH 1/1] hung_task: Explicitly report I/O wait state in log output

[Petr Mladek]

On Wed 2026-01-28 15:45:16, Aaron Tomlin wrote:
> Currently, the hung task reporting mechanism indiscriminately labels all
> TASK_UNINTERRUPTIBLE (D) tasks as "blocked", irrespective of whether they
> are awaiting I/O completion or kernel locking primitives. This ambiguity
> compels system administrators to manually inspect stack traces to discern
> whether the delay stems from an I/O wait (typically indicative of
> hardware or filesystem anomalies) or software contention. Such detailed
> analysis is not always immediately accessible to system administrators
> or support engineers.
> 
> To address this, this patch utilises the existing in_iowait field within
> struct task_struct to augment the failure report. If the task is blocked
> due to I/O (e.g., via io_schedule_prepare()), the log message is updated
> to explicitly state "blocked in I/O wait".
> 
> Examples:
>         - Standard Block: "INFO: task bash:123 blocked for more than 120
>           seconds".
> 
>         - I/O Block: "INFO: task dd:456 blocked in I/O wait for more than
>           120 seconds".
> 
> Accessing in_iowait is safe in this context. The detector holds
> rcu_read_lock() within check_hung_uninterruptible_tasks(), ensuring the
> task structure remains valid in memory.

This part is true.

> Furthermore, as the task is
> confirmed to be in a persistent TASK_UNINTERRUPTIBLE state, it cannot
> modify its own in_iowait flag, rendering the read operation stable and
> free from data races.

Strictly speaking, this is not true. IMHO, nothing prevents the task
from waking up in parallel. Just the chance is small.

I would say that the information will be valid in 99.99% of situations
because the message is printed only when the task has been stuck
in the state for a long time. A possible mistake should be
visible from the later printed backtrace.

> --- a/kernel/hung_task.c
> +++ b/kernel/hung_task.c
> @@ -250,8 +250,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
>  	if (sysctl_hung_task_warnings || hung_task_call_panic) {
>  		if (sysctl_hung_task_warnings > 0)
>  			sysctl_hung_task_warnings--;
> -		pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n",
> -		       t->comm, t->pid, (jiffies - t->last_switch_time) / HZ);
> +		pr_err("INFO: task %s:%d blocked %s for more than %ld seconds.\n",

s/blocked %s for/blocked%s for/

> +		       t->comm, t->pid, t->in_iowait ? "in I/O wait" : "",

and here: " in I/O wait".

Otherwise, it would print two spaces in the non-io case, like"

  "INFO: task bash:123 blocked  for more than 120 seconds"

> +		       (jiffies - t->last_switch_time) / HZ);
>  		pr_err("      %s %s %.*s\n",
>  			print_tainted(), init_utsname()->release,
>  			(int)strcspn(init_utsname()->version, " "),

Otherwise, it looks good.

Best Regards,
Petr

Re: [v2 PATCH 1/1] hung_task: Explicitly report I/O wait state in log output

[Lance Yang]

On 2026/2/2 23:14, Petr Mladek wrote:
> On Wed 2026-01-28 15:45:16, Aaron Tomlin wrote:
>> Currently, the hung task reporting mechanism indiscriminately labels all
>> TASK_UNINTERRUPTIBLE (D) tasks as "blocked", irrespective of whether they
>> are awaiting I/O completion or kernel locking primitives. This ambiguity
>> compels system administrators to manually inspect stack traces to discern
>> whether the delay stems from an I/O wait (typically indicative of
>> hardware or filesystem anomalies) or software contention. Such detailed
>> analysis is not always immediately accessible to system administrators
>> or support engineers.
>>
>> To address this, this patch utilises the existing in_iowait field within
>> struct task_struct to augment the failure report. If the task is blocked
>> due to I/O (e.g., via io_schedule_prepare()), the log message is updated
>> to explicitly state "blocked in I/O wait".
>>
>> Examples:
>>          - Standard Block: "INFO: task bash:123 blocked for more than 120
>>            seconds".
>>
>>          - I/O Block: "INFO: task dd:456 blocked in I/O wait for more than
>>            120 seconds".
>>
>> Accessing in_iowait is safe in this context. The detector holds
>> rcu_read_lock() within check_hung_uninterruptible_tasks(), ensuring the
>> task structure remains valid in memory.
> 
> This part is true.
> 
>> Furthermore, as the task is
>> confirmed to be in a persistent TASK_UNINTERRUPTIBLE state, it cannot
>> modify its own in_iowait flag, rendering the read operation stable and
>> free from data races.
> 
> Strictly speaking, this is not true. IMHO, nothing prevents the task
> from waking up in parallel. Just the chance is small.
> 
> I would say that the information will be valid in 99.99% of situations
> because the message is printed only when the task has been stuck
> in the state for a long time. A possible mistake should be
> visible from the later printed backtrace.
> 
>> --- a/kernel/hung_task.c
>> +++ b/kernel/hung_task.c
>> @@ -250,8 +250,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
>>   	if (sysctl_hung_task_warnings || hung_task_call_panic) {
>>   		if (sysctl_hung_task_warnings > 0)
>>   			sysctl_hung_task_warnings--;
>> -		pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n",
>> -		       t->comm, t->pid, (jiffies - t->last_switch_time) / HZ);
>> +		pr_err("INFO: task %s:%d blocked %s for more than %ld seconds.\n",
> 
> s/blocked %s for/blocked%s for/
> 
>> +		       t->comm, t->pid, t->in_iowait ? "in I/O wait" : "",
> 
> and here: " in I/O wait".
> 
> Otherwise, it would print two spaces in the non-io case, like"
> 
>    "INFO: task bash:123 blocked  for more than 120 seconds"

Good catch!

> 
>> +		       (jiffies - t->last_switch_time) / HZ);
>>   		pr_err("      %s %s %.*s\n",
>>   			print_tainted(), init_utsname()->release,
>>   			(int)strcspn(init_utsname()->version, " "),
> 
> Otherwise, it looks good.

Other than that, looks good to me.

Re: [v2 PATCH 1/1] hung_task: Explicitly report I/O wait state in log output

[Aaron Tomlin]

[-- Attachment #1: Type: text/plain, Size: 1673 bytes --]

On Mon, Feb 02, 2026 at 04:14:42PM +0100, Petr Mladek wrote:
> > Furthermore, as the task is
> > confirmed to be in a persistent TASK_UNINTERRUPTIBLE state, it cannot
> > modify its own in_iowait flag, rendering the read operation stable and
> > free from data races.
> 
> Strictly speaking, this is not true. IMHO, nothing prevents the task
> from waking up in parallel. Just the chance is small.
> 
> I would say that the information will be valid in 99.99% of situations
> because the message is printed only when the task has been stuck
> in the state for a long time. A possible mistake should be
> visible from the later printed backtrace.

Hi Petr,

Thank you for the review and the keen eye on the formatting.

You are entirely correct regarding the potential race; claiming the value
is strictly "stable" was imprecise. I have updated the commit message to
reflect that this is a diagnostic snapshot.

I also agree that we should not attempt to strictly prevent this race.

Theoretically, io_schedule_finish() could be called immediately after
"khungtaskd" checks the flag, rendering the output stale. However, strictly
preventing this would require blocking the waking task (e.g., within the
mutex unlock code path - see mutex_lock_io()) to inhibit the state change
during the scan. This seems entirely disproportionate for a "best effort"
diagnostic tool, especially given the probability of such a race is
negligible after a long hang, by default.

I have also adjusted the pr_err() format string to eliminate the potential
double space in the non-I/O case.

> Otherwise, it looks good.


Kind Regards,
-- 
Aaron Tomlin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [v2 PATCH 1/1] hung_task: Explicitly report I/O wait state in log output

[Aaron Tomlin]

[-- Attachment #1: Type: text/plain, Size: 409 bytes --]

On Thu, Jan 29, 2026 at 08:16:14AM +0900, Masami Hiramatsu wrote:
> Looks good to me.
> 
> Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>

Hi Masami,

Thank you.

I will apply your "Acked-by:" to the next version, which will also
incorporate the minor adjustment requested by Petr [1].


[1]: https://lore.kernel.org/lkml/aYC_Yr6SQVTUkyV6@pathway/ 

Kind regards
-- 
Aaron Tomlin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]