Is this RAID recoverable???

Is this RAID recoverable???

[Paul Richter]

Hi guys,

I got 8Tb on the line here, and I need to know if I am wasting my time
now, or what the next step is to recovery ...

Background:
RAID 6, lost a controller card and two drives with it, didn't know.
Some files were added in degraded state.
Problem with MB, lost a 3rd drive.
No writes of course, not enough drives to assemble raid.

Researched the net as much as possible, added this to wiki page:
http://en.wikipedia.org/wiki/Mdadm#Recovering_from_a_loss_of_raid_superblock

So the above is exactly what I did to try and recover the raid.

Present:
Running in degraded state, trying to mount and then will add remaining disks.

clop Desktop # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5]
[raid4] [multipath] [faulty]
md127 : active raid6 sdc1[0] sde1[3] sdf1[2] sdd1[1]
      7814037504 blocks super 1.2 level 6, 4096k chunk, algorithm 2
[6/4] [UUUU__]

md0 : active raid0 sdb1[0] sdj1[4] sdi1[3] sdh1[2] sdg1[1]
      312601600 blocks super 1.1 512k chunks

unused devices: <none>

clop Desktop # mdadm --detail /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Sun Jan  1 20:04:05 2012
     Raid Level : raid6
     Array Size : 7814037504 (7452.05 GiB 8001.57 GB)
  Used Dev Size : 1953509376 (1863.01 GiB 2000.39 GB)
   Raid Devices : 6
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun Jan  1 20:04:19 2012
          State : clean, degraded
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 4096K

           Name : clop:1  (local to host clop)
           UUID : 696be297:5056e6e5:86181fb8:7b323fd6
         Events : 2

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1
       2       8       81        2      active sync   /dev/sdf1
       3       8       65        3      active sync   /dev/sde1
       4       0        0        4      removed
       5       0        0        5      removed

This is what I get when I try and mount:

clop Desktop # mount /dev/md127 /media/
mount: wrong fs type, bad option, bad superblock on /dev/md127,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

clop Desktop # dmesg | tail
[   22.959022] ata11: EH complete
[   23.521403] ata12.00: configured for UDMA/133
[   23.521408] ata12: EH complete
[   23.573277] EXT4-fs (md0): re-mounted. Opts: commit=0
[   23.642304] EXT4-fs (sda1): re-mounted. Opts: commit=0
[   23.717975] EXT4-fs (sda2): re-mounted. Opts: commit=0
[   23.719060] EXT4-fs (sdl1): re-mounted. Opts: commit=0
[   31.945290] eth1: no IPv6 routers present
[  126.446097] libfcoe_device_notification: NETDEV_UNREGISTER lo
[  529.325204] EXT4-fs (md127): bad geometry: block count 1953512448
exceeds size of device (1953509376 blocks)


Have tried to get these numbers to change, nothing effects them.

Is this thing dead? Or what's my next step to recovery?

Many thanks!

Re: Is this RAID recoverable???

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

On Tue, 3 Jan 2012 10:41:04 +0400
Paul Richter <iamcomcy@gmail.com> wrote:

> Researched the net as much as possible, added this to wiki page:
> http://en.wikipedia.org/wiki/Mdadm#Recovering_from_a_loss_of_raid_superblock

Oh, so it was you who originally wrote those instructions? :)
See http://permalink.gmane.org/gmane.linux.raid/35538

And also you probably mean http://en.wikipedia.org/w/index.php?title=Mdadm&oldid=468952888#Recovering_from_a_loss_of_raid_superblock
because that part was nuked from the wikipedia page already.

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Is this RAID recoverable???

[Phil Turmel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Paul, Roman,

On 01/03/2012 04:37 AM, Roman Mamedov wrote:
> On Tue, 3 Jan 2012 10:41:04 +0400
> Paul Richter <iamcomcy@gmail.com> wrote:
> 
>> Researched the net as much as possible, added this to wiki page:
>> http://en.wikipedia.org/wiki/Mdadm#Recovering_from_a_loss_of_raid_superblock
> 
> Oh, so it was you who originally wrote those instructions? :)
> See http://permalink.gmane.org/gmane.linux.raid/35538

I was going to cite myself ...

> And also you probably mean http://en.wikipedia.org/w/index.php?title=Mdadm&oldid=468952888#Recovering_from_a_loss_of_raid_superblock
> because that part was nuked from the wikipedia page already.

As to the original question:  The odds are poor, but not zero.  If you
read my advice to Marcin, you know that some information from your
original array is needed.  Do you have any of it?

The array size error that mount reports suggests that you are close
to correct.  I suspect the original array was created with a different
data offset from the new array.  The default for that changed around
mdadm v2.6.9.

HTH,

Phil

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk8DAe4ACgkQBP+iHzflm3Ad+QCeJa7af9gammN2kGDFd+7qGMsM
rX0An0A0GXfJR3W4J0odv4LE8/rgsa+M
=8TYC
-----END PGP SIGNATURE-----

Re: Is this RAID recoverable???

[Paul Richter]

Hi Phil!
Thanks for throwing in! Do you have the older default values? I
suredon't. And how do you add offsets the the --create command?
mdadm --create --assume-clean /dev/md1 --chunk=4096
--level=6--raid-devices=6 /dev/sdc1 /dev/sdd1 /dev/sdf1 /dev/sde1
missingmissing
Is what I am working with now.
Many thanks!
This 8TB represents the last 8 years of my life on the net. Built
thisRAID to protect that 'investment'. :'-(
TIA

Paul
On Tue, Jan 3, 2012 at 5:26 PM, Phil Turmel <philip@turmel.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Paul, Roman,
>
> On 01/03/2012 04:37 AM, Roman Mamedov wrote:
>> On Tue, 3 Jan 2012 10:41:04 +0400
>> Paul Richter <iamcomcy@gmail.com> wrote:
>>
>>> Researched the net as much as possible, added this to wiki page:
>>> http://en.wikipedia.org/wiki/Mdadm#Recovering_from_a_loss_of_raid_superblock
>>
>> Oh, so it was you who originally wrote those instructions? :)
>> See http://permalink.gmane.org/gmane.linux.raid/35538
>
> I was going to cite myself ...
>
>> And also you probably mean http://en.wikipedia.org/w/index.php?title=Mdadm&oldid=468952888#Recovering_from_a_loss_of_raid_superblock
>> because that part was nuked from the wikipedia page already.
>
> As to the original question:  The odds are poor, but not zero.  If you
> read my advice to Marcin, you know that some information from your
> original array is needed.  Do you have any of it?
>
> The array size error that mount reports suggests that you are close
> to correct.  I suspect the original array was created with a different
> data offset from the new array.  The default for that changed around
> mdadm v2.6.9.
>
> HTH,
>
> Phil
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.18 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk8DAe4ACgkQBP+iHzflm3Ad+QCeJa7af9gammN2kGDFd+7qGMsM
> rX0An0A0GXfJR3W4J0odv4LE8/rgsa+M
> =8TYC
> -----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Is this RAID recoverable???

[Phil Turmel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Paul,

[Please avoid top-posting on kernel.org lists]

On 01/03/2012 09:22 AM, Paul Richter wrote:
> Hi Phil!
> Thanks for throwing in! Do you have the older default values? I
> suredon't. And how do you add offsets the the --create command?
> mdadm --create --assume-clean /dev/md1 --chunk=4096
> --level=6--raid-devices=6 /dev/sdc1 /dev/sdd1 /dev/sdf1 /dev/sde1
> missingmissing
> Is what I am working with now.
> Many thanks!

The "data offset" is not settable from the command line.  The modern
default is 1MB == 2048 sectors.  The prior default was something like
264 sectors.  To get it, you need to grab an older version of mdadm
(just for the --create operation).

> This 8TB represents the last 8 years of my life on the net. Built
> thisRAID to protect that 'investment'. :'-(

RAID != backup.  A common misconception.

> TIA

You're welcome.

Phil
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk8DEe0ACgkQBP+iHzflm3BVIwCfUvkQzKfrG4oFM2XDf4pVzfCR
EWgAnjPAsLVYkqZmol/WDhaDieEuWFw0
=8QkU
-----END PGP SIGNATURE-----

Re: Is this RAID recoverable???

[Phil Turmel]

Hi Paul,

[You top-posted again...]

Paul Richter <iamcomcy@gmail.com> wrote:

>So, I should DD back up the whole thing before trying to reconstruct.

The create operation with v2.6.9 isn't any more dangerous than the create operations already done.

>If the reconstruct is successful, then I build a new array and copy
>everything from the old to the new, right?

When/if the recreate succeeds, do a filesystem backup, recreate the array again, and restore.

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Re: Is this RAID recoverable???

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 939 bytes --]

On Tue, 3 Jan 2012 13:59:36 +0400
Paul Richter <iamcomcy@gmail.com> wrote:

> Yeah, that was me, based on the best info that could be found.
> 
> Perhaps, rather than just nuke that page, leaving everyone with nothing,
> you guys will write a correct one? That would likely be more useful.
> Clearly this problem comes up often enough that a page needs to be written.

The section was nuked not because of its inaccuracy, but simply because Wikipedia does not condone adding "HOWTO content" into it:
  
  http://en.wikipedia.org/wiki/Wikipedia:What_Wikipedia_is_not#NOTHOWTO

And not by me or any of linux-raid subscribers, but by someone from the Wikipedia community, I assume.

P.S. don't forget to use "Reply to All" when replying to the mailing list.

-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]