Re: Issue with Raid 10 super block failing

[Phil Turmel <philip@turmel.org>]

On 11/19/2012 11:44 AM, Drew Reusser wrote:
> On Mon, Nov 19, 2012 at 1:39 PM, Phil Turmel <philip@turmel.org> wrote:

[trim /]

>> So you've re-created the MD array.  That's bad.  Chunk size and Data
>> offset size and alignment defaults have changed in the past couple
>> years, so re-creating an array with a different mdadm version can cause
>> these problems.  You can also lose the original order of devices, with
>> similar consequences.
>>
> 
> Yes I did multiple creates to try to get the devices back together
> after mdadm --Fail commands.  I did not know about the assemble
> command yet and was following what "experts" were saying to try to
> recover failed superblock errors after a reboot (which is what errors
> I found).

The odds of success have dropped.  If you used "--assume-clean" *every*
time you used "--create", the odds are still greater than zero.
Otherwise, the odds your data is destroyed is *very* high.

>> (Side note:  there's various pieces of advice floating around the
>> internet on recovering a broken array that start with re-creating the
>> array.  It's horribly wrong, and only a last resort, and only after
>> recording all the details about the original array.)
>>
>> Unless you kept a copy of "mdadm --examine /dev/sd[abde]1" for the
>> original array, this will be difficult to debug further.  Your best
>> chance is to go back to the version of mdadm available when you first
>> built the system and recreate with that, trying the various device order
>> combinations.
>>
>> Don't attempt to mount to check for success.  First, use "fsck -n" to
>> non-destructively check the FS.  If that gives few errors, then you can
>> mount the FS.
>>
>> Phil
> 
> I don't have the original mdadm --examine as I never knew to keep a
> copy of it.  I created this array when I installed Mint on this server
> in August, so the the version I am running now is the same as the
> version on the pen drive I am booting from.  I know the disks were all
> the same.  I set them up intentionally so they would be identical.
> 
> here is the output of the fsck ..
> 
> mint mnt # fsck -n /dev/md0
> fsck from util-linux 2.20.1
> fsck: fsck.linux_raid_member: not found
> fsck: error 2 while executing fsck.linux_raid_member for /dev/md0

So, mount doesn't see it as an ext4 device at all.  Stop the array, and
scan each member for ext4 superblock magic:

for x in /dev/sd[abde]1 ; do echo $x ; \
dd if=$x bs=1M count=256 2>/dev/null | \
hexdump -C |grep '30  .\+  53 ef 0' ; done

Hopefully, each device will show one or more superblock candidates whose
offsets may help us decide which roles are which, and at what data offset.

Phil