Unusual RAID 1 recovery problem

Unusual RAID 1 recovery problem

[John Rowe]

Following a system reinstall (an upgrade from Scientific Linux 5.x  to
to 6.x), I had a RAID1 array that  I could start manually with:

> mdadm --assemble /dev/md0 /dev/sda4 /dev/sdb4

but would not start automatically on reboot. SL is a RedHat clone and
all partitions were of type "fd".

The above command worked fine and I could see all my data, but every
time I rebooted the RAID1 array wasn't there.

Encouraged by the reassuring words of the mdadm man page:

        --assume-clean
                      Tell mdadm that the array pre-existed and is known
        to be clean.  It can be  useful  when trying to recover from a
        major failure as you can be sure that no data will be affected
        unless you actually write to the array.
        
I tried:

> mdadm --create -l 1 -n 2 -assume-clean /dev/md0 /dev/sda4 /dev/sdb4

This worked, following the usual warning about how the partitions had
previously been part of an array. But now:

> mount -r /md0 /bob

refuses to do anything even if I try:

> mount -t ext2 -r /md0 /bob

I get an error message listing various possibilities such as "bad
superblock". dmesg tells me it can't find an ext2 file system
on /dev/md0


Clearly I had misunderstood the meaning of "you can be sure that no data
will be affected unless you actually write to the array" but I'm hoping
there is still a way of accessing this unaffected data.

Thanks.

John

Re: Unusual RAID 1 recovery problem

[Piergiorgio Sartor]

On Fri, May 10, 2013 at 06:31:15PM +0100, John Rowe wrote:
> Following a system reinstall (an upgrade from Scientific Linux 5.x  to
> to 6.x), I had a RAID1 array that  I could start manually with:
> 
> > mdadm --assemble /dev/md0 /dev/sda4 /dev/sdb4
> 
> but would not start automatically on reboot. SL is a RedHat clone and
> all partitions were of type "fd".
> 
> The above command worked fine and I could see all my data, but every
> time I rebooted the RAID1 array wasn't there.
> 
> Encouraged by the reassuring words of the mdadm man page:
> 
>         --assume-clean
>                       Tell mdadm that the array pre-existed and is known
>         to be clean.  It can be  useful  when trying to recover from a
>         major failure as you can be sure that no data will be affected
>         unless you actually write to the array.
>         
> I tried:
> 
> > mdadm --create -l 1 -n 2 -assume-clean /dev/md0 /dev/sda4 /dev/sdb4

Why oh why?
If assembly works, then the array is good,
likely the problem was in the initramfs or
the like.

> This worked, following the usual warning about how the partitions had
> previously been part of an array. But now:
> 
> > mount -r /md0 /bob
> 
> refuses to do anything even if I try:
> 
> > mount -t ext2 -r /md0 /bob
> 
> I get an error message listing various possibilities such as "bad
> superblock". dmesg tells me it can't find an ext2 file system
> on /dev/md0
> 
> 
> Clearly I had misunderstood the meaning of "you can be sure that no data
> will be affected unless you actually write to the array" but I'm hoping
> there is still a way of accessing this unaffected data.

Actually, you wrote to the array.
Using "create", you wrote a new superblock and, possibly,
a new "data offset".

Problem is that different versions of "mdadm" use different
offset for the data.
So now, likely, it could be your data starting point, were
ext2 should lie, is a bit more far than it was and ext2 is
not found anymore (obviously).

I think probably you can still recover you data, if the
offset could be somehow recovered.

Of course, it could be I'm wrong and something else failed.

bye,

pg

> 
> Thanks.
> 
> John
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 

piergiorgio

Re: Unusual RAID 1 recovery problem

[Tregaron Bayly]

On Fri, 2013-05-10 at 20:13 +0200, Piergiorgio Sartor wrote:
> > This worked, following the usual warning about how the partitions had
> > previously been part of an array. But now:
> > 
> > > mount -r /md0 /bob
> > 
> > refuses to do anything even if I try:
> > 
> > > mount -t ext2 -r /md0 /bob
> > 
> > I get an error message listing various possibilities such as "bad
> > superblock". dmesg tells me it can't find an ext2 file system
> > on /dev/md0
> > 
> > 
> > Clearly I had misunderstood the meaning of "you can be sure that no data
> > will be affected unless you actually write to the array" but I'm hoping
> > there is still a way of accessing this unaffected data.
> 
> Actually, you wrote to the array.
> Using "create", you wrote a new superblock and, possibly,
> a new "data offset".
> 
> Problem is that different versions of "mdadm" use different
> offset for the data.
> So now, likely, it could be your data starting point, were
> ext2 should lie, is a bit more far than it was and ext2 is
> not found anymore (obviously).
> 
> I think probably you can still recover you data, if the
> offset could be somehow recovered.

You are probably correct here.  The good news is that the filesystem is
still likely intact if you haven't done any operations on the new md0
device and you didn't choose a different raid metadata version when you
recreated it.  

Your best solution is to figure out which version of mdadm was used to
create this array originally and do your create again with that version.
If you don't know what version was used or you can't get hold of it then
you're going to have to figure out the true offset of the data.  This is
most easily accomplished by finding an old mdadm -E output from this
array.  If you don't have that you're going to have to go hunting on one
of your disks looking for a 0x55 0xaa boot signature that could indicate
an MBR.  Armed with the offset information you can either get the
development version of mdadm which Phil Turmel indicates has a
--data-offset option and create again or setup a loop device with that
offset on sda4 or sdb4 and mount it to get your data off to a new
device.

Good Luck,

Tregaron