Re: [RFC v2 0/1] ext4: fail fast on repeated buffer_head reads after IO failure

["Diangang Li" <lidiangang@bytedance.com>]

On 4/13/26 8:47 PM, Theodore Tso wrote:
> On Mon, Apr 13, 2026 at 02:24:59PM +0800, Diangang Li wrote:
>> From: Diangang Li <lidiangang@bytedance.com>
>>
>> A production system reported hung tasks blocked for 300s+ in ext4
>> buffer_head paths....
>>
>>    [Tue Mar 24 14:16:24 2026] blk_update_request: I/O error, dev sdi,
>>        sector 10704150288 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>>    [Tue Mar 24 14:16:25 2026] blk_update_request: I/O error, dev sdi,
>>        sector 10704488160 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
>>    [Tue Mar 24 14:16:26 2026] blk_update_request: I/O error, dev sdi,
>>        sector 10704382912 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
> 
> I wonder whether the ext4 layer is the right place to be handle this
> sort of issue.  For example, it could be handled by having a subsystem
> scanning dmesg (or by wiring up notifications so block device errors
> get sent to a userspace daemon), and when certain criteria is met, the
> machine is automatically sent to hardware operations to run
> diagnostics and (most likey) replace the failing disk.
> 
> It could also be handled in the driver or SCSI layer so the "fail
> fast" semantics are handled there, so that it supports all file
> systems, not just ext4.  The SCSI layer also has more information
> about the type of error; you might want to handle things like media
> errors differently from Fibre Channel or iSCSI timeouts (which might
> be something where "fast fast" is not appropriate).
> 
> By the time the error gets propagated up to the buffer head, we lose a
> lot of detail about why the error took place.  Also, in the long term
> we will hopefully be moving away from using buffer cache.
> 
>     		     	    	      	    - Ted

Hi Ted,

What about moving the fail-fast check into the buffer-head path 
(submit_bh_wbc) so it is not ext4-specific. We can update a BH_Read_EIO 
bit in end_bio_bh_io_sync, and add a per-bdev/per-partition sysfs knob 
for the retry window. That turns it into a generic guard for buffer-head 
users, and it naturally goes away as buffer-head usage shrinks.

We did think about doing this in the block layer (submit_bio) or in 
SCSI/NVMe, but a generic solution there seems to need a per-device table 
to cache the error LBAs. With buffer-head, we can keep the error state 
on the bh itself.

I also checked f2fs (no buffer-head). It tracks repeated EIOs on 
metadata/node pages to avoid infinite retry loops. How do you see that 
compared with a buffer-head retry window? Are either of these directions 
worth exploring further?

Thanks,
Diangang