big raid5 trouble

big raid5 trouble

[Czigola Gabor]

Hello all!

I'm not sure, that is this the right place to ask such question, but I'm 
in a big trouble with my RAID5 array.

My last chance to get any of the data back stored in this array, is to
"unspare" a spare disk, that is untouched since the removal.

I googled but can't find neither a documentation nor a specification where 
and how the spare flag is stored on a disk. Can you please help me?

Thanks a lot.
-- 
Czigola, Gabor

Re: big raid5 trouble

[berk walker]

Czigola Gabor wrote:

> Hello all!
>
> I'm not sure, that is this the right place to ask such question, but 
> I'm in a big trouble with my RAID5 array.
>
> My last chance to get any of the data back stored in this array, is to
> "unspare" a spare disk, that is untouched since the removal.
>
> I googled but can't find neither a documentation nor a specification 
> where and how the spare flag is stored on a disk. Can you please help me?
>
> Thanks a lot.

I don 't think that will help you, per se.  Why do you say it is your 
last chance? If you have n-1 disks OK, then you may be OK.  If not, a 
spare will not help.  Can you tell us more?
b-

Re: big raid5 trouble

[Czigola Gabor]

On Sun, 1 Jan 2006, berk walker wrote:

> I don 't think that will help you, per se.  Why do you say it is your last 
> chance? If you have n-1 disks OK, then you may be OK.  If not, a spare will 
> not help.  Can you tell us more?
> b-
>

So there is a RAID5 array with 4 disks. I had a long power failure, and 
during the boot sequence, I saw that the 4th disk got marked as non fresh. 
I started the process that should bring the 4th disk back in the array, 
but during this the 3th disk died. The 4th disk is now marked as spare.

Meanwhile a found the 4k superblock on the disk (strace mdadm -E), and I 
got - from the source of mdadm - the exact location of the spare flag. But 
I guess, I have to recalc the correct checksum of the new superblock.(?)

-- 
Czigola, Gabor

Re: big raid5 trouble

[berk walker]

Czigola Gabor wrote:

> On Sun, 1 Jan 2006, berk walker wrote:
>
>> I don 't think that will help you, per se.  Why do you say it is your 
>> last chance? If you have n-1 disks OK, then you may be OK.  If not, a 
>> spare will not help.  Can you tell us more?
>> b-
>>
>
> So there is a RAID5 array with 4 disks. I had a long power failure, 
> and during the boot sequence, I saw that the 4th disk got marked as 
> non fresh. I started the process that should bring the 4th disk back 
> in the array, but during this the 3th disk died. The 4th disk is now 
> marked as spare.
>
> Meanwhile a found the 4k superblock on the disk (strace mdadm -E), and 
> I got - from the source of mdadm - the exact location of the spare 
> flag. But I guess, I have to recalc the correct checksum of the new 
> superblock.(?)
>
The "spare" disk would be just that, spare, like a spare tire for a car, 
pressure is up, ready to go.  Each drive contained unique information, 
which could be reconstructed from the remaining n-1 drives.  You now 
have 1 drive "the spare" with no data, and an n-2  array.  You will 
probably lose data, though the 3rd disk might have had errors in an 
unused area.  You can FORCE the drives back into an array if things 
aren't too trashed.  I can not tell what is the safest way to do this, 
but someone here may. 

b-

Re: big raid5 trouble

[Czigola Gabor]

On Mon, 2 Jan 2006, berk walker wrote:

> The "spare" disk would be just that, spare, like a spare tire for a car, 
> pressure is up, ready to go.  Each drive contained unique information, which 
> could be reconstructed from the remaining n-1 drives.  You now have 1 drive 
> "the spare" with no data, and an n-2  array.  You will probably lose data, 
> though the 3rd disk might have had errors in an unused area.  You can FORCE 
> the drives back into an array if things aren't too trashed.  I can not tell 
> what is the safest way to do this, but someone here may. 
> b-

Yes, but the spare disk is just logically spare, because it was in the 
array just before everything got wrong. I mean the data is untouched on 
it. If I force it (with editing the raw disk superblock) to get a normal 
active disk, should it behave like before the cutoff, shouldn't it? Or 
does the RAID mechanism erase/change anything else on the disk that makes 
this process impossible?

-- 
Czigola Gabor

Re: big raid5 trouble

[Mitchell Laks]

On Monday 02 January 2006 09:46 am, Czigola Gabor wrote:

>
> Yes, but the spare disk is just logically spare, because it was in the
> array just before everything got wrong. I mean the data is untouched on
> it. If I force it (with editing the raw disk superblock) to get a normal
> active disk, should it behave like before the cutoff, shouldn't it? Or
> does the RAID mechanism erase/change anything else on the disk that makes
> this process impossible?

I am very interested in your recovery process having gone through it myself a 
number of times with hardware raid. 

I would also like to see how one can "set the flag" to accept the "spare" 
drive - into the array.

I hope the gurus here can tell us how to do it.

One thing you can do. Before you play with the data on your array, you can 
back up the individual hard drives to truely spare drives. Here you can use 
dd, or
dd with ignore error flags - see man page and google and dd_rescue man page or 
dd_rescue or 
dd_rescue with dd_rescue_help.

These amazing tools may even help with the "dead" drive. You can see what you 
can rescue. I was able to save some substantial data using those tools in the 
past. Beware that it may take a long time to  back up a severly damaged disk 
- thats why the tools were invented. 

Then you can try different techniques with your status "frozen".

It is this kind of headache that switched me to raid1 ...

Good Luck
Mitchell Laks