Subject: RFC: Read repair for md RAID1 after mirror read failures

Subject: RFC: Read repair for md RAID1 after mirror read failures

[G.W. Kant - Hunenet B.V.]

Hello list,

Recently I encountered an interesting failure mode while migrating data 
from an aging backup system.

The source Btrfs filesystem spanned several LVM logical volumes, each 
backed by an md RAID1 array. One of these logical volumes resided on a 
degraded RAID1 array, so I added a new logical volume on a new RAID1 
array and started migrating the data using:

btrfs device remove

During the migration, many read requests encountered unrecoverable read 
errors (UNC) on the remaining member of the degraded RAID1 array. The 
migration continued, which is exactly what I had hoped for, since only a 
small fraction of the media appeared to be affected.

This made me wonder whether md RAID1 has, or has ever considered, a read 
repair mechanism.

Consider the following situation:

mirror A:
read -> UNC

mirror B:
read -> OK

During a RAID1 read request, if md can satisfy the read from the 
alternate mirror, no data is lost. However, this also represents an 
opportunity to repair the degraded copy. Once a read has failed on one 
mirror, the array has effectively lost its ability to tolerate a second 
read failure for that logical block. This window persists until the 
affected block is rewritten by normal filesystem activity, which may 
never happen on cold archival data. Repairing the degraded copy 
immediately may restore full redundancy while a valid copy is still 
available. Writing the recovered block back to the the corresponding 
block on the failed mirror would give the drive an opportunity to 
recover that location, for example by successfully rewriting the sector 
or remapping it if the write cannot be verified.

In other words, the first successfully recovered read request could 
automatically become a repair opportunity. The repair could even be 
scheduled asynchronously, so the successful read is returned immediately 
while the rewrite is performed in the background. Unlike a periodic 
resync, this repair would be driven by an actual read failure, making it 
targeted rather than rewriting the entire mirror.

With today's 18–24 TB HDDs and backup/archive workloads, where data may 
remain unchanged for years, latent media degradation seems increasingly 
relevant. A successful read from the alternate mirror may be one of the 
last opportunities to refresh such a sector before it becomes 
permanently unreadable.

One advantage of such an approach is that it does not require md to 
decide between two conflicting copies. One mirror has already reported 
an unrecoverable read error, while the other has successfully 
reconstructed the requested block. The proposal only applies to this 
specific case.

Has this idea been discussed before, or is there a reason why md 
deliberately avoids this type of read repair?

Regards,

Dion Kant

Re: Subject: RFC: Read repair for md RAID1 after mirror read failures

[Abd-Alrhman Masalkhi]

Hi Dion,

On Wed, Jul 15, 2026 at 02:52 +0000, G. W. Kant wrote:
> Hello list,
>
> Recently I encountered an interesting failure mode while migrating data 
> from an aging backup system.
>
> The source Btrfs filesystem spanned several LVM logical volumes, each 
> backed by an md RAID1 array. One of these logical volumes resided on a 
> degraded RAID1 array, so I added a new logical volume on a new RAID1 
> array and started migrating the data using:
>
> btrfs device remove
>
> During the migration, many read requests encountered unrecoverable read 
> errors (UNC) on the remaining member of the degraded RAID1 array. The 
> migration continued, which is exactly what I had hoped for, since only a 
> small fraction of the media appeared to be affected.
>
> This made me wonder whether md RAID1 has, or has ever considered, a read 
> repair mechanism.
>
> Consider the following situation:
>
> mirror A:
> read -> UNC
>
> mirror B:
> read -> OK
>
> During a RAID1 read request, if md can satisfy the read from the 
> alternate mirror, no data is lost. However, this also represents an 
> opportunity to repair the degraded copy. Once a read has failed on one 
> mirror, the array has effectively lost its ability to tolerate a second 
> read failure for that logical block. This window persists until the 
> affected block is rewritten by normal filesystem activity, which may 
> never happen on cold archival data. Repairing the degraded copy 
> immediately may restore full redundancy while a valid copy is still 
> available. Writing the recovered block back to the the corresponding 
> block on the failed mirror would give the drive an opportunity to 
> recover that location, for example by successfully rewriting the sector 
> or remapping it if the write cannot be verified.
>
> In other words, the first successfully recovered read request could 
> automatically become a repair opportunity. The repair could even be 
> scheduled asynchronously, so the successful read is returned immediately 
> while the rewrite is performed in the background. Unlike a periodic 
> resync, this repair would be driven by an actual read failure, making it 
> targeted rather than rewriting the entire mirror.
>
Yes, md has had this for a long time. Look at fix_read_error() in
raid1.c. It is called from handle_read_error() on any failed read. It
reads from a healthy mirror and rewrites the bad region on the failing
device, giving the drive a chance to rewrite or remap the sector. If the
rewrite fails, it records a bad block. md does this synchronously under
a frozen array, so it is not a missing feature.

The likely reason you didn't see it is that your array was already
degraded, so there was no healthy in-array copy for fix_read_error() to
recover from. In your case, you were likely able to retrieve the data
due to btrfs level redundancy, and md can't repair across arrays.

> With today's 18–24 TB HDDs and backup/archive workloads, where data may 
> remain unchanged for years, latent media degradation seems increasingly 
> relevant. A successful read from the alternate mirror may be one of the 
> last opportunities to refresh such a sector before it becomes 
> permanently unreadable.
>
And Check/Repair is the right defense for cold archival data on large
drives.

> One advantage of such an approach is that it does not require md to 
> decide between two conflicting copies. One mirror has already reported 
> an unrecoverable read error, while the other has successfully 
> reconstructed the requested block. The proposal only applies to this 
> specific case.
>
> Has this idea been discussed before, or is there a reason why md 
> deliberately avoids this type of read repair?
>
> Regards,
>
> Dion Kant

-- 
Best Regards,
Abd-Alrhman

Re: Subject: RFC: Read repair for md RAID1 after mirror read failures

[G.W. Kant - Hunenet B.V.]

On 7/15/26 7:27 AM, Abd-Alrhman Masalkhi wrote:
>> In other words, the first successfully recovered read request could
>> automatically become a repair opportunity. The repair could even be
>> scheduled asynchronously, so the successful read is returned immediately
>> while the rewrite is performed in the background. Unlike a periodic
>> resync, this repair would be driven by an actual read failure, making it
>> targeted rather than rewriting the entire mirror.
>>
> Yes, md has had this for a long time. Look at fix_read_error() in
> raid1.c. It is called from handle_read_error() on any failed read. It
> reads from a healthy mirror and rewrites the bad region on the failing
> device, giving the drive a chance to rewrite or remap the sector. If the
> rewrite fails, it records a bad block. md does this synchronously under
> a frozen array, so it is not a missing feature.
> 
> The likely reason you didn't see it is that your array was already
> degraded, so there was no healthy in-array copy for fix_read_error() to
> recover from. In your case, you were likely able to retrieve the data
> due to btrfs level redundancy, and md can't repair across arrays.
> 
>> With today's 18–24 TB HDDs and backup/archive workloads, where data may
>> remain unchanged for years, latent media degradation seems increasingly
>> relevant. A successful read from the alternate mirror may be one of the
>> last opportunities to refresh such a sector before it becomes
>> permanently unreadable.
>>
> And Check/Repair is the right defense for cold archival data on large
> drives.

Thank you. fix_read_error() was exactly the piece I was looking for. I 
had missed that md already performs read repair when another healthy 
mirror is available.

My situation was indeed different because the array had already lost one 
member, so the successful read originated from Btrfs redundancy across 
another md RAID1 rather than from the surviving md mirror.

One question remains, though. check/repair periodically reads all 
sectors, but as far as I understand it, successfully read sectors are 
not rewritten. From a media aging perspective this is a different 
problem than recovering from a read error. Have there ever been 
discussions about a true media refresh pass that rewrites successfully 
read sectors to refresh long-lived magnetic recordings?

Best regards,
Dion Kant

Re: Subject: RFC: Read repair for md RAID1 after mirror read failures

[Yu Kuai]

Hi,

在 2026/7/15 15:10, G.W. Kant - Hunenet B.V. 写道:
> On 7/15/26 7:27 AM, Abd-Alrhman Masalkhi wrote:
>>> In other words, the first successfully recovered read request could
>>> automatically become a repair opportunity. The repair could even be
>>> scheduled asynchronously, so the successful read is returned immediately
>>> while the rewrite is performed in the background. Unlike a periodic
>>> resync, this repair would be driven by an actual read failure, making it
>>> targeted rather than rewriting the entire mirror.
>>>
>> Yes, md has had this for a long time. Look at fix_read_error() in
>> raid1.c. It is called from handle_read_error() on any failed read. It
>> reads from a healthy mirror and rewrites the bad region on the failing
>> device, giving the drive a chance to rewrite or remap the sector. If the
>> rewrite fails, it records a bad block. md does this synchronously under
>> a frozen array, so it is not a missing feature.
>>
>> The likely reason you didn't see it is that your array was already
>> degraded, so there was no healthy in-array copy for fix_read_error() to
>> recover from. In your case, you were likely able to retrieve the data
>> due to btrfs level redundancy, and md can't repair across arrays.
>>
>>> With today's 18–24 TB HDDs and backup/archive workloads, where data may
>>> remain unchanged for years, latent media degradation seems increasingly
>>> relevant. A successful read from the alternate mirror may be one of the
>>> last opportunities to refresh such a sector before it becomes
>>> permanently unreadable.
>>>
>> And Check/Repair is the right defense for cold archival data on large
>> drives.
> Thank you. fix_read_error() was exactly the piece I was looking for. I
> had missed that md already performs read repair when another healthy
> mirror is available.
>
> My situation was indeed different because the array had already lost one
> member, so the successful read originated from Btrfs redundancy across
> another md RAID1 rather than from the surviving md mirror.
>
> One question remains, though. check/repair periodically reads all
> sectors, but as far as I understand it, successfully read sectors are
> not rewritten. From a media aging perspective this is a different
> problem than recovering from a read error. Have there ever been
> discussions about a true media refresh pass that rewrites successfully
> read sectors to refresh long-lived magnetic recordings?

Perhaps this is what you looking for:

[RFC v2 0/5] md/raid1: introduce a new sync action to repair badblocks - 
Zheng Qixing 
<https://lore.kernel.org/all/20260203061259.609206-1-zhengqixing@huaweicloud.com/>

However, AFAIK, Qixing no longer work in this area and there will not be 
new version.

>
> Best regards,
> Dion Kant

-- 
Thanks,
Kuai