Re: Grub-install, superblock corrupted/erased and other animals

[NeilBrown <neilb@suse.de>]

On Wed, 27 Jul 2011 14:16:52 +0200 Aaron Scheiner <blue@aquarat.za.net> wrote:

> Hi
> 
> My original message was sent (and rejected) 4 days ago because it was
> in HTML (whoops). Here's my original e-mail with updates :
> 
> I've got an Ubuntu machine with an mdadm RAID array on level 6. The
> RAID-6 array consists of 10 drives and has been running for about 3
> years now. I recently upgraded the drives in the array from 1TB units
> to 2TB units.
> 
> The drive on which the OS sat died a few days ago, so I installed a
> new OS drive and then installed Ubuntu Server on it.
> On reboot the machine hung on a black screen with a white flashing
> cursor. So I went back into the Ubuntu Setup and installed grub to all
> of the drives in the raid array (except two) [wow, this was such a
> stupid move].

So you still have two with valid superblocks?  (or did before you started
re-creating).  Do you have a copy of the "mdadm --examine" of those?  It
would be helpful.

> 
> I then rebooted the machine and it successfully booted into Ubuntu
> Server. I set about restoring the configuration for the raid array...
> only to be given the message "No Superblock Found" (more or less).
> Each element in the array was used directly by mdadm (so /dev/sda, not
> /dev/sda1).
> 
> I see that the superblock is stored within the MBR region on the drive
> (which is 512bytes from the start of the disk), which would explain
> why the superblocks were destroyed. I haven't been able to find
> anything regarding a backup superblock (does something like this
> exist?).
> 
> I have now started using a script that tries to re-create the array by
> running through the various permutations available... it takes roughly
> 2.5 seconds per permutation/iteration and there are just over 40 000
> possibilities. The script tests for a valid array by trying to mount
> the array as read only (it's an XFS file system). I somehow doubt that
> it will mount even when the correct combination of disks is found.
> [UPDATE] : It never mounted.

Odd...  Possibly you have a newer mdadm which uses a different "Data
offset".  The "mdadm --examine" of the 2 drives that didn't get corrupted
would help confirm that.

> 
> So... I have an idea... The array has a hole bunch of movie files on
> it and I have exact copies of some of them on another raid array. So I
> was thinking that if I searched for the start of one of those files on
> the scrambled array, I could work out the order of the disks by
> searching forward until I found a change. I could then compare the
> changed area (probably 128KB/the chunk size forward) with the file I
> have and see where that chunk lies in the file, thereby working out
> the order.
> [UPDATE] : Seeing as the array never mounted, I have proceeded with
> this idea. I took samples of the start of the video file and provided
> them to Scalpel as needles for recovery. After two days of searching,
> Scalpel located the starts of the various segments in the raid array
> (I re-created the raid array with the drives in random order). I then
> carved (using dd) 3MBs out of the raid array that contains all the
> samples handed to scalpel originally (plus a bit more).
> 
> So now I have to find segments of the start of the intact file in the
> carved out data from the raid array.
> 
> It would be really useful if I knew the layout of the array :
> 
> If the chunk size of the array is 128KB, does that mean that the file
> I carved will be divided up into segments of contiguous data, each
> 128KBs in length ? or does it mean that the length of contiguous data
> will be 16KB ( 128 KB / 8 drives ) ?

128KB per device, so the first alternative.

> 
> Do these segments follow on from each other without interruption or is
> there some other data in-between (like metadata? I'm not sure where
> that resides).

That depends on how XFS lays out the data.  It will probably be mostly
contiguous, but no guarantees.


> 
> Any explanation of the structure of a raid6 array would be greatly
> appreciated, as well as any other advice (tools, tips, etc).

The stripes start "Data Offset" from the beginning of the device, which I
think is 1MB with recent mdadm, but something like 64K with earlier mdadm.

The first few stripes are:

 Q  0  1  2  3  4  5  6  7  P
 8  9 10 11 12 13 14 15  P  Q
17 18 19 20 21 22 23  P  Q 16

Where 'P' is xor parity, Q is GF parity, and N is chunk number N.  Each chunk
is 128KB (if that is your chunk size).

This pattern repeats after 10 stripes.

good luck,
NeilBrown


> 
> Thanks :)
> 
> Aaron (24)
> South Africa
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html