RAID 5 crash, is there any way to recover some data ?

RAID 5 crash, is there any way to recover some data ?

[Sevrin Robstad]

I *had* a RAID 5 consisting of 6 x 200GB drives. After a power failure, 
my motherboard failed and I replaced it with some old crap I had.
After some lockups on this platform I suddenly had more than one disk 
marked as bad.

And then, after some googling around - I tried "mdadm --assemble 
--force" ... It seemed fine when rebuilding, but after some time the 
hardware failed again..
Now my raid was GONE.

 I have tried some "mdadm --build" things, but I cannot get any data out 
of it, it stops on "no Ext3 filesystem found" ...

Is there *any* way for me to recover at least some of the data ?


Sevrin Robstad,
Norway

Re: RAID 5 crash, is there any way to recover some data ?

[Mike Hardy]

Sevrin Robstad wrote:
> I *had* a RAID 5 consisting of 6 x 200GB drives. After a power failure,
> my motherboard failed and I replaced it with some old crap I had.
> After some lockups on this platform I suddenly had more than one disk
> marked as bad.
> 
> And then, after some googling around - I tried "mdadm --assemble
> --force" ... It seemed fine when rebuilding, but after some time the
> hardware failed again..
> Now my raid was GONE.
> 
> I have tried some "mdadm --build" things, but I cannot get any data out
> of it, it stops on "no Ext3 filesystem found" ...
> 
> Is there *any* way for me to recover at least some of the data ?


I feel your pain, I've been there.

What you want to do is determine which 5 drives out of your 6 drive
array have the most up-to-date data. You can do this by looking at the
output of mdadm -E <partition> ('mdadm -E /dev/sda1' or similar) and
seeing which ones were updated most recently.

These 5 drives are now you're most important drives ever, to get your
data back.

The trick is to actually create an entirely new array, using those five
drives in the same order they were in your previous array, with the
keyword 'missing' in place of where ever the sixth drive would go.

This will create an array that won't re-sync since it's missing a drive
and can't make parity, right? But the layout will be the exact same as
your old array, so when you start it, the old data should be there, and
you should be able to run a fsck on your filesystem and make sure your
data is there.

At this point, you should back everything up if you haven't already :-)

Finally, add your sixth drive to the array so it will fill the missing
slot, and you'll have redundancy again.

For example, a command that would work for create would be:

mdadm --create -l 5 -n 6 /dev/md3 /dev/sda1 /dev/sdb1 /dev/sdc1 \
	missing /dev/sde1 /dev/sdf1

It seems scary since you are creating a new array, but with that missing
slot, all you're doing is creating new superblocks, so it's fairly safe.

-Mike

Re: RAID 5 crash, is there any way to recover some data ?

[Sevrin Robstad]

Sevrin Robstad wrote:

> Mike Hardy wrote:
>
>>>I *had* a RAID 5 consisting of 6 x 200GB drives. After a power failure,
>>>my motherboard failed and I replaced it with some old crap I had.
>>>After some lockups on this platform I suddenly had more than one disk
>>>marked as bad.
>>>
>>>And then, after some googling around - I tried "mdadm --assemble
>>>--force" ... It seemed fine when rebuilding, but after some time the
>>>hardware failed again..
>>>Now my raid was GONE.
>>>
>>>I have tried some "mdadm --build" things, but I cannot get any data out
>>>of it, it stops on "no Ext3 filesystem found" ...
>>>
>>>Is there *any* way for me to recover at least some of the data ?
>>>    
>>>
>>I feel your pain, I've been there.
>>
>>What you want to do is determine which 5 drives out of your 6 drive
>>array have the most up-to-date data. You can do this by looking at the
>>output of mdadm -E <partition> ('mdadm -E /dev/sda1' or similar) and
>>seeing which ones were updated most recently.
>>
>>These 5 drives are now you're most important drives ever, to get your
>>data back.
>>
>>The trick is to actually create an entirely new array, using those five
>>drives in the same order they were in your previous array, with the
>>keyword 'missing' in place of where ever the sixth drive would go.
>>
>>This will create an array that won't re-sync since it's missing a drive
>>and can't make parity, right? But the layout will be the exact same as
>>your old array, so when you start it, the old data should be there, and
>>you should be able to run a fsck on your filesystem and make sure your
>>data is there.
>>
>>At this point, you should back everything up if you haven't already :-)
>>
>>Finally, add your sixth drive to the array so it will fill the missing
>>slot, and you'll have redundancy again.
>>
>>For example, a command that would work for create would be:
>>
>>mdadm --create -l 5 -n 6 /dev/md3 /dev/sda1 /dev/sdb1 /dev/sdc1 \
>>	missing /dev/sde1 /dev/sdf1
>>
>>It seems scary since you are creating a new array, but with that missing
>>slot, all you're doing is creating new superblocks, so it's fairly safe.
>>
>>  
>>
> Thanks for a quick reply.. . Will this work if I already have 
> (...stupid me...) done a "mdadm --create -l 5 -n 6 /dev/xx " *without* 
> the missing one - it has already done a rebuild once.
> I have done this now, and if I do a "fdisk /dev/md0" I get that 
> there's no partitions on the array... :(
>
> Sevrin

Re: RAID 5 crash, is there any way to recover some data ?

[Sevrin Robstad]

Is there not any way for me to recover my data....

As I said, i have already rebuilt the raid with wrong disc setup... Now 
i have built the raid with the right setup, and one missing disk.
But there's not any ext3 partition on it anymore. Is there any way to 
search trough the md0 device and find some of the data ???


Sevrin