best effort rebuild with 1 failed, 1 marginal disks in raid5?

best effort rebuild with 1 failed, 1 marginal disks in raid5?

[Brian Ristuccia]

I have a 3 disk raid5 array which recently suffered catastrophic failure of
one disk. While the array was degraded, one more disk failed out of the
array due to a bad block. Investigation with badblocks reveals the marginal
disk has only a handful of bad blocks.

What I'd like to do is start the array with the good disk and the marginal
disk, add a 3rd known good disk, and rebuild the array as best as possible.
Then I'd fail the marginal disk and install another good disk.  Finally, I'd
use the list from badblocks, debugfs, some algebra, and some guesswork to
determine which files I need to restore from backups. Since there's only a
few bad blocks, I suspect it'll be just a few files from tape and perhaps
fishing some stuff out of lost+found if any directories got damaged.

I am able to force the marginal disk back into the array with the '-f'
option to mdadm --assemble. When I add a 3rd good disk back into the set, it
begins a rebuild. But when it gets to one of the bad blocks on the marginal
disk, that disk is removed from the array and I'm left with a non-working
array and the rebuild stops.

Is there any way to tell the kernel raid resync stuff to make a best effort
for the rebuild, writing empty or bogus data for the bad blocks? This
filesystem is fairly large, so if I can avoid a complete restore from
backup it would save a lot of time. Also, it would make it vastly easier to
recover the files created between the most recent backup and now, which
would otherwise be quite laborious because currently I have to manually
reactivate the array every time my tar or dump stumbes on a file or
directory with a bad block. 

Note that I've already attempted the trick of trying to clear bad blocks by
writing to them similar to what's documented at
http://smartmontools.sourceforge.net/BadBlockHowTo.txt without much success.
I get I/O errors on write to these blocks also.

-- 
Brian Ristuccia

Re: best effort rebuild with 1 failed, 1 marginal disks in raid5?

[David Greaves]

Brian Ristuccia wrote:

>I have a 3 disk raid5 array which recently suffered catastrophic failure of
>one disk. While the array was degraded, one more disk failed out of the
>array due to a bad block. Investigation with badblocks reveals the marginal
>disk has only a handful of bad blocks.
>
>What I'd like to do is start the array with the good disk and the marginal
>disk, add a 3rd known good disk, and rebuild the array as best as possible.
>Then I'd fail the marginal disk and install another good disk.
>
Well, first of all - don't do anything without verifying it first.

There's a good chance you can recover quite some data if you don't screw 
it up by 'having a go' (I speak from experience :) )

Instead of your plan, can I suggest you try to duplicate the 
disk/partition showing a few bad blocks onto a known good disk.
Then forcefully reassemble the array, then add another good disk for 
reconstruction.

Useful tools include dd_rhelp and dd_rescue

Take a look in the archives as this kinda hit me recently.
http://marc.theaimsgroup.com/?l=linux-raid&m=110025421113782&w=2

David

RE: best effort rebuild with 1 failed, 1 marginal disks in raid5?

[Guy]

When you look at the archive, make sure you keep clicking "next in thread"
to see the complete story.  It did end well!

Guy

-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of David Greaves
Sent: Sunday, November 28, 2004 11:43 AM
To: Brian Ristuccia
Cc: linux-raid@vger.kernel.org
Subject: Re: best effort rebuild with 1 failed, 1 marginal disks in raid5?

Brian Ristuccia wrote:

>I have a 3 disk raid5 array which recently suffered catastrophic failure of
>one disk. While the array was degraded, one more disk failed out of the
>array due to a bad block. Investigation with badblocks reveals the marginal
>disk has only a handful of bad blocks.
>
>What I'd like to do is start the array with the good disk and the marginal
>disk, add a 3rd known good disk, and rebuild the array as best as possible.
>Then I'd fail the marginal disk and install another good disk.
>
Well, first of all - don't do anything without verifying it first.

There's a good chance you can recover quite some data if you don't screw 
it up by 'having a go' (I speak from experience :) )

Instead of your plan, can I suggest you try to duplicate the 
disk/partition showing a few bad blocks onto a known good disk.
Then forcefully reassemble the array, then add another good disk for 
reconstruction.

Useful tools include dd_rhelp and dd_rescue

Take a look in the archives as this kinda hit me recently.
http://marc.theaimsgroup.com/?l=linux-raid&m=110025421113782&w=2

David
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html