Re: Subject: RFC: Read repair for md RAID1 after mirror read failures

[Abd-Alrhman Masalkhi <abd.masalkhi@gmail.com>]

Hi Dion,

On Wed, Jul 15, 2026 at 02:52 +0000, G. W. Kant wrote:
> Hello list,
>
> Recently I encountered an interesting failure mode while migrating data 
> from an aging backup system.
>
> The source Btrfs filesystem spanned several LVM logical volumes, each 
> backed by an md RAID1 array. One of these logical volumes resided on a 
> degraded RAID1 array, so I added a new logical volume on a new RAID1 
> array and started migrating the data using:
>
> btrfs device remove
>
> During the migration, many read requests encountered unrecoverable read 
> errors (UNC) on the remaining member of the degraded RAID1 array. The 
> migration continued, which is exactly what I had hoped for, since only a 
> small fraction of the media appeared to be affected.
>
> This made me wonder whether md RAID1 has, or has ever considered, a read 
> repair mechanism.
>
> Consider the following situation:
>
> mirror A:
> read -> UNC
>
> mirror B:
> read -> OK
>
> During a RAID1 read request, if md can satisfy the read from the 
> alternate mirror, no data is lost. However, this also represents an 
> opportunity to repair the degraded copy. Once a read has failed on one 
> mirror, the array has effectively lost its ability to tolerate a second 
> read failure for that logical block. This window persists until the 
> affected block is rewritten by normal filesystem activity, which may 
> never happen on cold archival data. Repairing the degraded copy 
> immediately may restore full redundancy while a valid copy is still 
> available. Writing the recovered block back to the the corresponding 
> block on the failed mirror would give the drive an opportunity to 
> recover that location, for example by successfully rewriting the sector 
> or remapping it if the write cannot be verified.
>
> In other words, the first successfully recovered read request could 
> automatically become a repair opportunity. The repair could even be 
> scheduled asynchronously, so the successful read is returned immediately 
> while the rewrite is performed in the background. Unlike a periodic 
> resync, this repair would be driven by an actual read failure, making it 
> targeted rather than rewriting the entire mirror.
>
Yes, md has had this for a long time. Look at fix_read_error() in
raid1.c. It is called from handle_read_error() on any failed read. It
reads from a healthy mirror and rewrites the bad region on the failing
device, giving the drive a chance to rewrite or remap the sector. If the
rewrite fails, it records a bad block. md does this synchronously under
a frozen array, so it is not a missing feature.

The likely reason you didn't see it is that your array was already
degraded, so there was no healthy in-array copy for fix_read_error() to
recover from. In your case, you were likely able to retrieve the data
due to btrfs level redundancy, and md can't repair across arrays.

> With today's 18–24 TB HDDs and backup/archive workloads, where data may 
> remain unchanged for years, latent media degradation seems increasingly 
> relevant. A successful read from the alternate mirror may be one of the 
> last opportunities to refresh such a sector before it becomes 
> permanently unreadable.
>
And Check/Repair is the right defense for cold archival data on large
drives.

> One advantage of such an approach is that it does not require md to 
> decide between two conflicting copies. One mirror has already reported 
> an unrecoverable read error, while the other has successfully 
> reconstructed the requested block. The proposal only applies to this 
> specific case.
>
> Has this idea been discussed before, or is there a reason why md 
> deliberately avoids this type of read repair?
>
> Regards,
>
> Dion Kant

-- 
Best Regards,
Abd-Alrhman