raid6 recovery with read errors

raid6 recovery with read errors

[Matthias Urlichs]

Hi,

I have a rather large RAID6 array where multiple disks have developed
read errors. The problem is that the RAID6 is built from LVM-mapped disks,
which (I think) isolated the MD driver from write errors. Thus it thought
that a re-write of the bad sectors succeeded. WRONG. Thus the bad disks
were never unmapped.

The problem is that yesterday, three of these TByte disks failed in
_exactly_ the same 1k-sized spot. So, no more RAID6. :-(

So, how do I get the data back?

I've copied the individual partitions with ddrescue, which conveniently
left me a log file pointing to the sectors which need to be recovered.
However, I'm sure that there's no way to tell the kernel about individual
"bad spots".

Is there a standalone program that can do that?

-- 
Matthias Urlichs

Re: raid6 recovery with read errors

[Matthias Urlichs]

On Mon, 03 Aug 2009 08:36:06 +0000, Matthias Urlichs wrote:

> [ block-level restoration of RAID6 data ]
> Is there a standalone program that can do that?

Seems that the answer is "no", at least as far as anybody on this list is 
concerned.

*Sigh* So I'll have to write one, it seems.

[PATCH] Re: raid6 recovery with read errors

[Matthias Urlichs]

>> [ block-level restoration of RAID6 data ] Is there a standalone program
>> that can do that?
> 
> Seems that the answer is "no", at least as far as anybody on this list
> is concerned.
> 
> *Sigh* So I'll have to write one, it seems.

The code is available from http://netz.smurf.noris.de/git/mdadm.git 
(branch "repair"). This is off mdadm's 3.1 development branch; the bugfix 
changes since 3.0 have been merged in.

Restoration of RAID5 and RAID6 partitions is tested, given a workable 
badblock list (i.e. ddrescue's log). Restoration of RAID6 and purging 
unrecoverable ranges (plus restoring the parity sectors) is untested.

The test suite includes a helper program which can find bad parity or 
inconsistent RAID1. I didn't add an option to mdadm to do that, because 
I'm lazy :-P besides, the kernel can do that already.

Also included are some changes in the test suite which ensure that the 
test suite works on a machine which has udev running -- or loopback or 
raid drivers in use. :-/

Usage: mdadm --repair-blocks=0:bad-sda1.log,1:bad-sdb1.log /dev/sd[a-d]1
(The numbers correspond to the disk numbers within the RAID, not to the 
n'th non-option argument.) Specifying a UUID instead of a bunch of 
devices should also work, but is untested.

Any feedback is of course welcome.

Now for the difficult part -- how to do automagic remapping. I tend to 
think that extending the staging/cowloop (copy-on-write) driver to be a 
coeloop (copy-on-error) driver would be a good idea. (It should also be 
possible to force a copy of the RAID superblocks, and overwrite that with 
some sort of cowloop metadata on the original disk, so that the kernel 
doesn't mount the underlying device with its obsolete blocks. :-/ )

TODO: (a) containers. Somebody else please uncomment and re-enable the 
code in Repair.c; (b) testing of RAID1 and uuid-etc.-based RAID selection,
(c) auto-selection among 'identical' RAID partitions, if scanning finds 
more than one (the idea is to choose the partition that's _not_ on the 
device where the superblock says it should be, assuming identical event 
numbers).

I haven't tested this on my production system yet -- that's the next 
step. But the test suite says it should work. Famous last words. :-P