[RFC PATCH 0/1] ext4: fail fast on repeated metadata reads after IO failure

[RFC PATCH 0/1] ext4: fail fast on repeated metadata reads after IO failure

[Diangang Li]

From: Diangang Li <lidiangang@bytedance.com>

A production system reported hung tasks blocked for 300s+ in ext4 metadata
lookup paths. Hung task reports were accompanied by disk IO errors, but
profiling showed that most individual reads completed (or failed) within
10s, with the worst case around 60s.

At the same time, we observed a high repeat rate to the same disk LBAs.
The repeated reads frequently showed seconds-level latency and ended with
IO errors, e.g.:

  [Tue Mar 24 14:16:24 2026] blk_update_request: I/O error, dev sdi,
      sector 10704150288 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
  [Tue Mar 24 14:16:25 2026] blk_update_request: I/O error, dev sdi,
      sector 10704488160 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
  [Tue Mar 24 14:16:26 2026] blk_update_request: I/O error, dev sdi,
      sector 10704382912 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0

We also sampled repeated-LBA latency histograms on /dev/sdi and saw that
the same error-prone LBAs were re-submitted many times with ~1-4s latency:

  LBA 10704488160 (count=22): 1-2s: 20, 2-4s: 2
  LBA 10704382912 (count=21): 1-2s: 20, 2-4s: 1
  LBA 10704150288 (count=21): 1-2s: 19, 2-4s: 2

Root cause
==========

ext4 metadata reads commonly use buffer_head caching and serialize IO via
BH_Lock (lock_buffer). When a read fails, the buffer remains !Uptodate.
With multiple threads concurrently accessing the same metadata block, each
waiter wakes up after the previous owner drops BH_Lock, then submits the
same read again and waits again. This makes the latency grow linearly with
the number of contending threads, leading to 300s+ hung tasks.

The failing IOs are repeatedly issued to the same LBA. The observed 1s+
per-IO latency is likely from device-side retry/error recovery. On SCSI the
driver typically retries reads several times (e.g. 5 retries in our
environment), so a single filesystem submission can easily accumulate 5s+
delay before failing. When multiple threads then re-submit the same failing
metadata read and serialize on BH_Lock, the delay is amplified into 300s+
hung tasks.

Similar behavior exists for other devices (e.g. NVMe with multiple internal
retries).

Example hung stacks:

  INFO: task toutiao.infra.t:3760933 blocked for more than 327 seconds.
  Call Trace:
   __schedule
   io_schedule
   __wait_on_bit_lock
   bh_uptodate_or_lock
   __read_extent_tree_block
   ext4_find_extent
   ext4_ext_map_blocks
   ext4_map_blocks
   ext4_getblk
   ext4_bread
   __ext4_read_dirblock
   dx_probe
   ext4_htree_fill_tree
   ext4_readdir
   iterate_dir
   ksys_getdents64

  INFO: task toutiao.infra.t:2724456 blocked for more than 327 seconds.
  Call Trace:
   __schedule
   io_schedule
   __wait_on_bit_lock
   ext4_read_bh_lock
   ext4_bread
   __ext4_read_dirblock
   htree_dirblock_to_tree
   ext4_htree_fill_tree
   ext4_readdir
   iterate_dir
   ksys_getdents64

Approach
========

Remember read failures on buffer_head and fail fast for ext4 metadata reads
once a buffer has already seen a read failure. Clear the flag on successful
read/write completion so the buffer can recover if the error is transient.

Note that ext4 read-ahead uses ext4_read_bh_nowait(), so it does not set
the failure flag and remains best-effort.

Patch summary
=============

  1) Add BH_Read_EIO buffer_head state bit and helpers.
  2) Clear BH_Read_EIO on successful read/write completion.
  3) In ext4 metadata reads, if BH_Read_EIO is already set (and not
     BH_Write_EIO), fail fast instead of re-submitting the same failing
     read. On read failure, set BH_Read_EIO.

Diangang Li (1):
  ext4: fail fast on repeated metadata reads after IO failure

 fs/buffer.c                 |  2 ++
 fs/ext4/super.c             | 12 +++++++++++-
 include/linux/buffer_head.h |  2 ++
 3 files changed, 15 insertions(+), 1 deletion(-)

-- 
2.39.5

[RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure

[Diangang Li]

From: Diangang Li <lidiangang@bytedance.com>

ext4 metadata reads serialize on BH_Lock (lock_buffer). If the read fails,
the buffer remains !Uptodate. With concurrent callers, each waiter can
retry the same failing read after the previous holder drops BH_Lock. This
amplifies device retry latency and may trigger hung tasks.

In the normal read path the block driver already performs its own retries.
Once the retries keep failing, re-submitting the same metadata read from
the filesystem just amplifies the latency by serializing waiters on
BH_Lock.

Remember read failures on buffer_head and fail fast for ext4 metadata reads
once a buffer has already failed to read. Clear the flag on successful
read/write completion so the buffer can recover. ext4 read-ahead uses
ext4_read_bh_nowait(), so it does not set the failure flag and remains
best-effort.

Example hung stacks:

  INFO: task toutiao.infra.t:3760933 blocked for more than 327 seconds.
  Call Trace:
   __schedule
   io_schedule
   __wait_on_bit_lock
   bh_uptodate_or_lock
   __read_extent_tree_block
   ext4_find_extent
   ext4_ext_map_blocks
   ext4_map_blocks
   ext4_getblk
   ext4_bread
   __ext4_read_dirblock
   dx_probe
   ext4_htree_fill_tree
   ext4_readdir
   iterate_dir
   ksys_getdents64

  INFO: task toutiao.infra.t:2724456 blocked for more than 327 seconds.
  Call Trace:
   __schedule
   io_schedule
   __wait_on_bit_lock
   ext4_read_bh_lock
   ext4_bread
   __ext4_read_dirblock
   htree_dirblock_to_tree
   ext4_htree_fill_tree
   ext4_readdir
   iterate_dir
   ksys_getdents64

Signed-off-by: Diangang Li <lidiangang@bytedance.com>
Reviewed-by: Fengnan Chang <changfengnan@bytedance.com>
---
 fs/buffer.c                 |  2 ++
 fs/ext4/super.c             | 12 +++++++++++-
 include/linux/buffer_head.h |  2 ++
 3 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/fs/buffer.c b/fs/buffer.c
index 2d2e3ecec6b2b..b41d54b8b1f4d 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -145,6 +145,7 @@ static void __end_buffer_read_notouch(struct buffer_head *bh, int uptodate)
 {
 	if (uptodate) {
 		set_buffer_uptodate(bh);
+		clear_buffer_read_io_error(bh);
 	} else {
 		/* This happens, due to failed read-ahead attempts. */
 		clear_buffer_uptodate(bh);
@@ -167,6 +168,7 @@ void end_buffer_write_sync(struct buffer_head *bh, int uptodate)
 {
 	if (uptodate) {
 		set_buffer_uptodate(bh);
+		clear_buffer_read_io_error(bh);
 	} else {
 		buffer_io_error(bh, ", lost sync page write");
 		mark_buffer_write_io_error(bh);
diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index 781c083000c2e..89a99851864a0 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -198,7 +198,13 @@ int ext4_read_bh(struct buffer_head *bh, blk_opf_t op_flags,
 {
 	BUG_ON(!buffer_locked(bh));
 
+	if (!buffer_write_io_error(bh) && buffer_read_io_error(bh)) {
+		unlock_buffer(bh);
+		return -EIO;
+	}
+
 	if (ext4_buffer_uptodate(bh)) {
+		clear_buffer_read_io_error(bh);
 		unlock_buffer(bh);
 		return 0;
 	}
@@ -206,8 +212,12 @@ int ext4_read_bh(struct buffer_head *bh, blk_opf_t op_flags,
 	__ext4_read_bh(bh, op_flags, end_io, simu_fail);
 
 	wait_on_buffer(bh);
-	if (buffer_uptodate(bh))
+	if (buffer_uptodate(bh)) {
+		clear_buffer_read_io_error(bh);
 		return 0;
+	}
+	if (!buffer_write_io_error(bh))
+		set_buffer_read_io_error(bh);
 	return -EIO;
 }
 
diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h
index b16b88bfbc3e7..be8bedcde379e 100644
--- a/include/linux/buffer_head.h
+++ b/include/linux/buffer_head.h
@@ -29,6 +29,7 @@ enum bh_state_bits {
 	BH_Delay,	/* Buffer is not yet allocated on disk */
 	BH_Boundary,	/* Block is followed by a discontiguity */
 	BH_Write_EIO,	/* I/O error on write */
+	BH_Read_EIO,	/* I/O error on read */
 	BH_Unwritten,	/* Buffer is allocated on disk but not written */
 	BH_Quiet,	/* Buffer Error Prinks to be quiet */
 	BH_Meta,	/* Buffer contains metadata */
@@ -132,6 +133,7 @@ BUFFER_FNS(Async_Write, async_write)
 BUFFER_FNS(Delay, delay)
 BUFFER_FNS(Boundary, boundary)
 BUFFER_FNS(Write_EIO, write_io_error)
+BUFFER_FNS(Read_EIO, read_io_error)
 BUFFER_FNS(Unwritten, unwritten)
 BUFFER_FNS(Meta, meta)
 BUFFER_FNS(Prio, prio)
-- 
2.39.5

Re: [RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure

[Andreas Dilger]

On Mar 25, 2026, at 03:33, Diangang Li <diangangli@gmail.com> wrote:
> 
> From: Diangang Li <lidiangang@bytedance.com>
> 
> ext4 metadata reads serialize on BH_Lock (lock_buffer). If the read fails,
> the buffer remains !Uptodate. With concurrent callers, each waiter can
> retry the same failing read after the previous holder drops BH_Lock. This
> amplifies device retry latency and may trigger hung tasks.
> 
> In the normal read path the block driver already performs its own retries.
> Once the retries keep failing, re-submitting the same metadata read from
> the filesystem just amplifies the latency by serializing waiters on
> BH_Lock.
> 
> Remember read failures on buffer_head and fail fast for ext4 metadata reads
> once a buffer has already failed to read. Clear the flag on successful
> read/write completion so the buffer can recover. ext4 read-ahead uses
> ext4_read_bh_nowait(), so it does not set the failure flag and remains
> best-effort.

Not that the patch is bad, but if the BH_Read_EIO flag is set on a buffer
and it prevents other tasks from reading that block again, how would the
buffer ever become Uptodate to clear the flag?  There isn't enough state
in a 1-bit flag to have any kind of expiry and later retry.

Cheers, Andreas

Re: [RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure

[Diangang Li]

Hi Andreas,

BH_Read_EIO is cleared on successful read or write.

In practice bad blocks are typically repaired/remapped on write, so we
expect recovery after a successful rewrite. If the block is never
rewritten, repeatedly issuing the same failing read does not help.

We clear the flag on successful reads so the buffer can recover 
immediately if the error was transient. Since read-ahead reads are not 
blocked, a later successful read-ahead will clear the flag and allow 
subsequent synchronous readers to proceed normally.

Best,
Diangang

On 3/25/26 6:15 PM, Andreas Dilger wrote:
> On Mar 25, 2026, at 03:33, Diangang Li <diangangli@gmail.com> wrote:
>>
>> From: Diangang Li <lidiangang@bytedance.com>
>>
>> ext4 metadata reads serialize on BH_Lock (lock_buffer). If the read fails,
>> the buffer remains !Uptodate. With concurrent callers, each waiter can
>> retry the same failing read after the previous holder drops BH_Lock. This
>> amplifies device retry latency and may trigger hung tasks.
>>
>> In the normal read path the block driver already performs its own retries.
>> Once the retries keep failing, re-submitting the same metadata read from
>> the filesystem just amplifies the latency by serializing waiters on
>> BH_Lock.
>>
>> Remember read failures on buffer_head and fail fast for ext4 metadata reads
>> once a buffer has already failed to read. Clear the flag on successful
>> read/write completion so the buffer can recover. ext4 read-ahead uses
>> ext4_read_bh_nowait(), so it does not set the failure flag and remains
>> best-effort.
> 
> Not that the patch is bad, but if the BH_Read_EIO flag is set on a buffer
> and it prevents other tasks from reading that block again, how would the
> buffer ever become Uptodate to clear the flag?  There isn't enough state
> in a 1-bit flag to have any kind of expiry and later retry.
> 
> Cheers, Andreas

Re: [RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure

[Zhang Yi]

Hi, Diangang,

On 3/25/2026 7:13 PM, Diangang Li wrote:
> Hi Andreas,
> 
> BH_Read_EIO is cleared on successful read or write.

I think what Andreas means is, since you modified the ext4_read_bh() 
interface, if the bh to be read already has the Read_EIO flag set, then 
subsequent read operations through this interface will directly return 
failure without issuing a read I/O. At the same time, because its state 
is also not uptodate, for an existing block, a write request will not be 
issued either. How can we clear this Read_EIO flag? IIRC, relying solely 
on ext4_read_bh_nowait() doesn't seem sufficient to achieve this.

Thanks,
Yi.

> 
> In practice bad blocks are typically repaired/remapped on write, so we
> expect recovery after a successful rewrite. If the block is never
> rewritten, repeatedly issuing the same failing read does not help.
> 
> We clear the flag on successful reads so the buffer can recover
> immediately if the error was transient. Since read-ahead reads are not
> blocked, a later successful read-ahead will clear the flag and allow
> subsequent synchronous readers to proceed normally.
> 
> Best,
> Diangang
> 
> On 3/25/26 6:15 PM, Andreas Dilger wrote:
>> On Mar 25, 2026, at 03:33, Diangang Li <diangangli@gmail.com> wrote:
>>>
>>> From: Diangang Li <lidiangang@bytedance.com>
>>>
>>> ext4 metadata reads serialize on BH_Lock (lock_buffer). If the read fails,
>>> the buffer remains !Uptodate. With concurrent callers, each waiter can
>>> retry the same failing read after the previous holder drops BH_Lock. This
>>> amplifies device retry latency and may trigger hung tasks.
>>>
>>> In the normal read path the block driver already performs its own retries.
>>> Once the retries keep failing, re-submitting the same metadata read from
>>> the filesystem just amplifies the latency by serializing waiters on
>>> BH_Lock.
>>>
>>> Remember read failures on buffer_head and fail fast for ext4 metadata reads
>>> once a buffer has already failed to read. Clear the flag on successful
>>> read/write completion so the buffer can recover. ext4 read-ahead uses
>>> ext4_read_bh_nowait(), so it does not set the failure flag and remains
>>> best-effort.
>>
>> Not that the patch is bad, but if the BH_Read_EIO flag is set on a buffer
>> and it prevents other tasks from reading that block again, how would the
>> buffer ever become Uptodate to clear the flag?  There isn't enough state
>> in a 1-bit flag to have any kind of expiry and later retry.
>>
>> Cheers, Andreas
> 

Re: [RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure

[Matthew Wilcox]

On Wed, Mar 25, 2026 at 04:15:42AM -0600, Andreas Dilger wrote:
> On Mar 25, 2026, at 03:33, Diangang Li <diangangli@gmail.com> wrote:
> > 
> > From: Diangang Li <lidiangang@bytedance.com>
> > 
> > ext4 metadata reads serialize on BH_Lock (lock_buffer). If the read fails,
> > the buffer remains !Uptodate. With concurrent callers, each waiter can
> > retry the same failing read after the previous holder drops BH_Lock. This
> > amplifies device retry latency and may trigger hung tasks.
> > 
> > In the normal read path the block driver already performs its own retries.
> > Once the retries keep failing, re-submitting the same metadata read from
> > the filesystem just amplifies the latency by serializing waiters on
> > BH_Lock.
> > 
> > Remember read failures on buffer_head and fail fast for ext4 metadata reads
> > once a buffer has already failed to read. Clear the flag on successful
> > read/write completion so the buffer can recover. ext4 read-ahead uses
> > ext4_read_bh_nowait(), so it does not set the failure flag and remains
> > best-effort.
> 
> Not that the patch is bad, but if the BH_Read_EIO flag is set on a buffer
> and it prevents other tasks from reading that block again, how would the
> buffer ever become Uptodate to clear the flag?  There isn't enough state
> in a 1-bit flag to have any kind of expiry and later retry.

I've been thinking about this problem too, albeit from a folio read
perspective, not from a buffer_head read perspective.  You're quite
right that one bit isn't enough.  The solution I was considering but
haven't implemented yet was to tell all the current waiters that
the IO has failed, but not set any kind of permanent error flag.

I was thinking about starting with this:

+++ b/include/linux/wait_bit.h
@@ -10,6 +10,7 @@
 struct wait_bit_key {
        unsigned long           *flags;
        int                     bit_nr;
+       int                     error;
        unsigned long           timeout;
 };


and then adding/changing various APIs to allow an error to be passed in
and noticed by the woken task.

With this change, the thundering herd all wake up, see the error and
return immediately instead of each submitting their own I/O.  New reads
will retry the read, but each will only be held up for a maximum of
their own timeout.

Re: [RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure

[changfengnan]

> From: "Zhang Yi"<yizhang089@gmail.com>
> Date:  Wed, Mar 25, 2026, 22:27
> Subject:  Re: [RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure
> To: "Diangang Li"<lidiangang@bytedance.com>, "Andreas Dilger"<adilger@dilger.ca>, "Diangang Li"<diangangli@gmail.com>
> Cc: <tytso@mit.edu>, <linux-ext4@vger.kernel.org>, <linux-fsdevel@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <changfengnan@bytedance.com>
> Hi, Diangang,
> 
> On 3/25/2026 7:13 PM, Diangang Li wrote:
> > Hi Andreas,
> > 
> > BH_Read_EIO is cleared on successful read or write.
> 
> I think what Andreas means is, since you modified the ext4_read_bh() 
> interface, if the bh to be read already has the Read_EIO flag set, then 
> subsequent read operations through this interface will directly return 
> failure without issuing a read I/O. At the same time, because its state

IMO, we first need to reach a consensus on whether we can expect a
retry to succeed after a read failure. 
Given that current SCSI and NVMe drivers already perform multiple
retries for I/O errors.
IMO, this depends on the specific error. If the block layer returns
BLK_STS_RESOURCE or BLK_STS_AGAIN, we can retry; however, if
it returns BLK_STS_MEDIUM or BLK_STS_IOERR, there is no need to retry.
For scenarios requiring a retry, we should also wait for a certain time
window before retrying.

Thanks.
Fengnan.

> is also not uptodate, for an existing block, a write request will not be 
> issued either. How can we clear this Read_EIO flag? IIRC, relying solely 
> on ext4_read_bh_nowait() doesn't seem sufficient to achieve this.
> 
> Thanks,
> Yi.
> 
> > 
> > In practice bad blocks are typically repaired/remapped on write, so we
> > expect recovery after a successful rewrite. If the block is never
> > rewritten, repeatedly issuing the same failing read does not help.
> > 
> > We clear the flag on successful reads so the buffer can recover
> > immediately if the error was transient. Since read-ahead reads are not
> > blocked, a later successful read-ahead will clear the flag and allow
> > subsequent synchronous readers to proceed normally.
> > 
> > Best,
> > Diangang
> > 
> > On 3/25/26 6:15 PM, Andreas Dilger wrote:
> >> On Mar 25, 2026, at 03:33, Diangang Li <diangangli@gmail.com> wrote:
> >>>
> >>> From: Diangang Li <lidiangang@bytedance.com>
> >>>
> >>> ext4 metadata reads serialize on BH_Lock (lock_buffer). If the read fails,
> >>> the buffer remains !Uptodate. With concurrent callers, each waiter can
> >>> retry the same failing read after the previous holder drops BH_Lock. This
> >>> amplifies device retry latency and may trigger hung tasks.
> >>>
> >>> In the normal read path the block driver already performs its own retries.
> >>> Once the retries keep failing, re-submitting the same metadata read from
> >>> the filesystem just amplifies the latency by serializing waiters on
> >>> BH_Lock.
> >>>
> >>> Remember read failures on buffer_head and fail fast for ext4 metadata reads
> >>> once a buffer has already failed to read. Clear the flag on successful
> >>> read/write completion so the buffer can recover. ext4 read-ahead uses
> >>> ext4_read_bh_nowait(), so it does not set the failure flag and remains
> >>> best-effort.
> >>
> >> Not that the patch is bad, but if the BH_Read_EIO flag is set on a buffer
> >> and it prevents other tasks from reading that block again, how would the
> >> buffer ever become Uptodate to clear the flag?  There isn't enough state
> >> in a 1-bit flag to have any kind of expiry and later retry.
> >>
> >> Cheers, Andreas
> >
> 

Re: [RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure

[Diangang Li]

Hi, Yi,

Thanks. Yes, for existing metadata blocks ext4 is read-modify-write, so 
without a successful read (Uptodate) there is no write path to update 
that block.

In the case we're seeing, the read keeps failing (repeated I/O errors on 
the same LBA), so the write never has a chance to run either. Given 
that, would it make sense (as Fengnan suggested) to treat persistent 
media errors (e.g. MEDIUM ERROR / IO ERROR) as non-retryable at the 
filesystem level, i.e. keep failing fast for that block? That would 
avoid the BH_Lock thundering herd and prevent hung tasks.

Thanks,
Diangang

On 3/25/26 10:27 PM, Zhang Yi wrote:
> Hi, Diangang,
> 
> On 3/25/2026 7:13 PM, Diangang Li wrote:
>> Hi Andreas,
>>
>> BH_Read_EIO is cleared on successful read or write.
> 
> I think what Andreas means is, since you modified the ext4_read_bh() 
> interface, if the bh to be read already has the Read_EIO flag set, then 
> subsequent read operations through this interface will directly return 
> failure without issuing a read I/O. At the same time, because its state 
> is also not uptodate, for an existing block, a write request will not be 
> issued either. How can we clear this Read_EIO flag? IIRC, relying solely 
> on ext4_read_bh_nowait() doesn't seem sufficient to achieve this.
> 
> Thanks,
> Yi.
> 
>>
>> In practice bad blocks are typically repaired/remapped on write, so we
>> expect recovery after a successful rewrite. If the block is never
>> rewritten, repeatedly issuing the same failing read does not help.
>>
>> We clear the flag on successful reads so the buffer can recover
>> immediately if the error was transient. Since read-ahead reads are not
>> blocked, a later successful read-ahead will clear the flag and allow
>> subsequent synchronous readers to proceed normally.
>>
>> Best,
>> Diangang
>>
>> On 3/25/26 6:15 PM, Andreas Dilger wrote:
>>> On Mar 25, 2026, at 03:33, Diangang Li <diangangli@gmail.com> wrote:
>>>>
>>>> From: Diangang Li <lidiangang@bytedance.com>
>>>>
>>>> ext4 metadata reads serialize on BH_Lock (lock_buffer). If the read 
>>>> fails,
>>>> the buffer remains !Uptodate. With concurrent callers, each waiter can
>>>> retry the same failing read after the previous holder drops BH_Lock. 
>>>> This
>>>> amplifies device retry latency and may trigger hung tasks.
>>>>
>>>> In the normal read path the block driver already performs its own 
>>>> retries.
>>>> Once the retries keep failing, re-submitting the same metadata read 
>>>> from
>>>> the filesystem just amplifies the latency by serializing waiters on
>>>> BH_Lock.
>>>>
>>>> Remember read failures on buffer_head and fail fast for ext4 
>>>> metadata reads
>>>> once a buffer has already failed to read. Clear the flag on successful
>>>> read/write completion so the buffer can recover. ext4 read-ahead uses
>>>> ext4_read_bh_nowait(), so it does not set the failure flag and remains
>>>> best-effort.
>>>
>>> Not that the patch is bad, but if the BH_Read_EIO flag is set on a 
>>> buffer
>>> and it prevents other tasks from reading that block again, how would the
>>> buffer ever become Uptodate to clear the flag?  There isn't enough state
>>> in a 1-bit flag to have any kind of expiry and later retry.
>>>
>>> Cheers, Andreas
>>
>

Re: [RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure

[Zhang Yi]

On 3/26/2026 3:42 PM, Diangang Li wrote:
> Hi, Yi,
> 
> Thanks. Yes, for existing metadata blocks ext4 is read-modify-write, so 
> without a successful read (Uptodate) there is no write path to update 
> that block.
> 
> In the case we're seeing, the read keeps failing (repeated I/O errors on 
> the same LBA), so the write never has a chance to run either. Given 
> that, would it make sense (as Fengnan suggested) to treat persistent 
> media errors (e.g. MEDIUM ERROR / IO ERROR) as non-retryable at the 
> filesystem level, i.e. keep failing fast for that block? That would 
> avoid the BH_Lock thundering herd and prevent hung tasks.
> 

FYI, AFAICT, while this approach makes sense in theory, it actually
faces challenges in fault recovery. This is because these error codes
are not always reliable (especially BLK_STS_IOERR). In some scenarios
where reliability requirements are not very high, customers might not
immediately notice these errors due to transient faults on some storage
devices(such as some network storage scenarios), and these errors might
resolve themselves after a certain period of time. However, after this,
we have to perform some heavy-weight operations, such as stopping
services and remounting the file system, to recover our services. I
believe there will definitely be customers who will complain about
this.

Thanks,
Yi.

> Thanks,
> Diangang
> 
> On 3/25/26 10:27 PM, Zhang Yi wrote:
>> Hi, Diangang,
>>
>> On 3/25/2026 7:13 PM, Diangang Li wrote:
>>> Hi Andreas,
>>>
>>> BH_Read_EIO is cleared on successful read or write.
>>
>> I think what Andreas means is, since you modified the ext4_read_bh() 
>> interface, if the bh to be read already has the Read_EIO flag set, then 
>> subsequent read operations through this interface will directly return 
>> failure without issuing a read I/O. At the same time, because its state 
>> is also not uptodate, for an existing block, a write request will not be 
>> issued either. How can we clear this Read_EIO flag? IIRC, relying solely 
>> on ext4_read_bh_nowait() doesn't seem sufficient to achieve this.
>>
>> Thanks,
>> Yi.
>>
>>>
>>> In practice bad blocks are typically repaired/remapped on write, so we
>>> expect recovery after a successful rewrite. If the block is never
>>> rewritten, repeatedly issuing the same failing read does not help.
>>>
>>> We clear the flag on successful reads so the buffer can recover
>>> immediately if the error was transient. Since read-ahead reads are not
>>> blocked, a later successful read-ahead will clear the flag and allow
>>> subsequent synchronous readers to proceed normally.
>>>
>>> Best,
>>> Diangang
>>>
>>> On 3/25/26 6:15 PM, Andreas Dilger wrote:
>>>> On Mar 25, 2026, at 03:33, Diangang Li <diangangli@gmail.com> wrote:
>>>>>
>>>>> From: Diangang Li <lidiangang@bytedance.com>
>>>>>
>>>>> ext4 metadata reads serialize on BH_Lock (lock_buffer). If the read 
>>>>> fails,
>>>>> the buffer remains !Uptodate. With concurrent callers, each waiter can
>>>>> retry the same failing read after the previous holder drops BH_Lock. 
>>>>> This
>>>>> amplifies device retry latency and may trigger hung tasks.
>>>>>
>>>>> In the normal read path the block driver already performs its own 
>>>>> retries.
>>>>> Once the retries keep failing, re-submitting the same metadata read 
>>>>> from
>>>>> the filesystem just amplifies the latency by serializing waiters on
>>>>> BH_Lock.
>>>>>
>>>>> Remember read failures on buffer_head and fail fast for ext4 
>>>>> metadata reads
>>>>> once a buffer has already failed to read. Clear the flag on successful
>>>>> read/write completion so the buffer can recover. ext4 read-ahead uses
>>>>> ext4_read_bh_nowait(), so it does not set the failure flag and remains
>>>>> best-effort.
>>>>
>>>> Not that the patch is bad, but if the BH_Read_EIO flag is set on a 
>>>> buffer
>>>> and it prevents other tasks from reading that block again, how would the
>>>> buffer ever become Uptodate to clear the flag?  There isn't enough state
>>>> in a 1-bit flag to have any kind of expiry and later retry.
>>>>
>>>> Cheers, Andreas
>>>
>>
> 
> 

Re: [RFC 1/1] ext4: fail fast on repeated metadata reads after IO failure

[Diangang Li]

On 3/25/26 11:06 PM, Matthew Wilcox wrote:
> On Wed, Mar 25, 2026 at 04:15:42AM -0600, Andreas Dilger wrote:
>> On Mar 25, 2026, at 03:33, Diangang Li <diangangli@gmail.com> wrote:
>>>
>>> From: Diangang Li <lidiangang@bytedance.com>
>>>
>>> ext4 metadata reads serialize on BH_Lock (lock_buffer). If the read fails,
>>> the buffer remains !Uptodate. With concurrent callers, each waiter can
>>> retry the same failing read after the previous holder drops BH_Lock. This
>>> amplifies device retry latency and may trigger hung tasks.
>>>
>>> In the normal read path the block driver already performs its own retries.
>>> Once the retries keep failing, re-submitting the same metadata read from
>>> the filesystem just amplifies the latency by serializing waiters on
>>> BH_Lock.
>>>
>>> Remember read failures on buffer_head and fail fast for ext4 metadata reads
>>> once a buffer has already failed to read. Clear the flag on successful
>>> read/write completion so the buffer can recover. ext4 read-ahead uses
>>> ext4_read_bh_nowait(), so it does not set the failure flag and remains
>>> best-effort.
>>
>> Not that the patch is bad, but if the BH_Read_EIO flag is set on a buffer
>> and it prevents other tasks from reading that block again, how would the
>> buffer ever become Uptodate to clear the flag?  There isn't enough state
>> in a 1-bit flag to have any kind of expiry and later retry.
> 
> I've been thinking about this problem too, albeit from a folio read
> perspective, not from a buffer_head read perspective.  You're quite
> right that one bit isn't enough.  The solution I was considering but
> haven't implemented yet was to tell all the current waiters that
> the IO has failed, but not set any kind of permanent error flag.
> 
> I was thinking about starting with this:
> 
> +++ b/include/linux/wait_bit.h
> @@ -10,6 +10,7 @@
>   struct wait_bit_key {
>          unsigned long           *flags;
>          int                     bit_nr;
> +       int                     error;
>          unsigned long           timeout;
>   };
> 
> 
> and then adding/changing various APIs to allow an error to be passed in
> and noticed by the woken task.
> 
> With this change, the thundering herd all wake up, see the error and
> return immediately instead of each submitting their own I/O.  New reads
> will retry the read, but each will only be held up for a maximum of
> their own timeout.

Hi Matthew and all,

Thanks. The idea of waking the current waiters with an error makes a lot 
of sense.

I’ve been considering a smaller change on the buffer_head side that 
might get most of the benefit without touching the generic wait_bit 
APIs. The idea is to tell whether taking BH_Lock required waiting. If we 
had to wait, and the buffer is already marked with 
buffer_read_io_error(), then just return -EIO and don’t submit another 
read. If we got the lock without waiting, still submit the read. That 
should stop the thundering herd from reissuing the same failing IO.

Another option is a simple retry window. After a read failure, don’t
retry for some period of time, and size that window by error type. For
persistent media errors (e.g. MEDIUM ERROR, repeated IO ERROR) the 
window could be effectively infinite, while for transient cases (e.g. 
few IO ERROR, BLK_STS_RESOURCE) it could be small.

Any opinions on these two approaches, or other ideas for this problem?

Thanks,
Diangang