Raid 5 Recovery

Raid 5 Recovery

[Erik Boettcher]

Hello everyone, I'm trying to recover a raid-5 array that suffered
from a powerloss (and possibly de-seating of a SATA card that 3 out of
the 5 drives were on).
As far as I can tell, the drives themselves are still okay.
At first this is what happened when I tried mounting it from a livecd:

md: Autodetecting RAID arrays.
Bad version number 0.88 on sda3
md: sda3 does not have a valid v0.90 superblock, not importing!
Bad version number 0.88 on sdb3
md: sdb3 does not have a valid v0.90 superblock, not importing!
Bad version number 0.88 on sdc3
md: sdc3 does not have a valid v0.90 superblock, not importing!
md: Scanned 5 and added 2 devices.
md: autorun ...
md: considering sde3 ...
md:  adding sde3 ...
md:  adding sdd3 ...
md: created md0
md: bind<sdd3>
md: bind<sde3>
md: running: <sde3><sdd3>
raid5: device sde3 operational as raid disk 2
raid5: device sdd3 operational as raid disk 1
raid5: not enough operational devices for md0 (3/5 failed)
RAID5 conf printout:
 --- rd:5 wd:2
 disk 1, o:1, dev:sdd3
 disk 2, o:1, dev:sde3
raid5: failed to run raid set md0
md: pers->run() failed ...
md: do_md_run() returned -5
md: md0 stopped.
md: unbind<sde3>
md: export_rdev(sde3)
md: unbind<sdd3>
md: export_rdev(sdd3)
md: ... autorun DONE.

At this point I sought help on ##linux on freenode, running this:
mack ~ # mdadm -D /dev/md0
mdadm: md device /dev/md0 does not appear to be active.
mack ~ # mdadm --assemble /dev/md0 --scan
mdadm: superblock on /dev/sdc3 doesn't match others - assembly aborted
mack ~ # mdadm -A --force /dev/md0
mdadm: superblock on /dev/sdc3 doesn't match others - assembly aborted
mack ~ # mdadm -A --force /dev/md0 /dev/sd{a,b,c,d,e}3
mdadm: superblock on /dev/sdd3 doesn't match others - assembly aborted

At this point I said WOAH -- time to stop with the freenode help --
I'm hoping I didn't cause too much more damage there...

I'm trying to get this array up and any help would be greatly
appreciated -- I don't want to do anything more to write to the drive
without expert advice.
Many thanks,
Erik Boettcher

Re: Raid 5 Recovery

[Redeeman]

On Sun, 2008-12-28 at 01:21 -0500, Erik Boettcher wrote:
> Hello everyone, I'm trying to recover a raid-5 array that suffered
> from a powerloss (and possibly de-seating of a SATA card that 3 out of
> the 5 drives were on).
> As far as I can tell, the drives themselves are still okay.
> At first this is what happened when I tried mounting it from a livecd:
> 
> md: Autodetecting RAID arrays.
> Bad version number 0.88 on sda3
> md: sda3 does not have a valid v0.90 superblock, not importing!
> Bad version number 0.88 on sdb3
> md: sdb3 does not have a valid v0.90 superblock, not importing!
> Bad version number 0.88 on sdc3
> md: sdc3 does not have a valid v0.90 superblock, not importing!
> md: Scanned 5 and added 2 devices.
> md: autorun ...
> md: considering sde3 ...
> md:  adding sde3 ...
> md:  adding sdd3 ...
> md: created md0
> md: bind<sdd3>
> md: bind<sde3>
> md: running: <sde3><sdd3>
> raid5: device sde3 operational as raid disk 2
> raid5: device sdd3 operational as raid disk 1
> raid5: not enough operational devices for md0 (3/5 failed)
> RAID5 conf printout:
>  --- rd:5 wd:2
>  disk 1, o:1, dev:sdd3
>  disk 2, o:1, dev:sde3
> raid5: failed to run raid set md0
> md: pers->run() failed ...
> md: do_md_run() returned -5
> md: md0 stopped.
> md: unbind<sde3>
> md: export_rdev(sde3)
> md: unbind<sdd3>
> md: export_rdev(sdd3)
> md: ... autorun DONE.
> 
> At this point I sought help on ##linux on freenode, running this:
> mack ~ # mdadm -D /dev/md0
> mdadm: md device /dev/md0 does not appear to be active.
> mack ~ # mdadm --assemble /dev/md0 --scan
> mdadm: superblock on /dev/sdc3 doesn't match others - assembly aborted
> mack ~ # mdadm -A --force /dev/md0
> mdadm: superblock on /dev/sdc3 doesn't match others - assembly aborted
> mack ~ # mdadm -A --force /dev/md0 /dev/sd{a,b,c,d,e}3
> mdadm: superblock on /dev/sdd3 doesn't match others - assembly aborted
> 

Im by no means an expert in this, but i think you need to force an
assembly of all the disks..

but, wait for the opinions of other more knowledgable people in this
area...


> At this point I said WOAH -- time to stop with the freenode help --
> I'm hoping I didn't cause too much more damage there...
> 
> I'm trying to get this array up and any help would be greatly
> appreciated -- I don't want to do anything more to write to the drive
> without expert advice.
> Many thanks,
> Erik Boettcher
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid 5 Recovery

[Dylan Distasio]

I am far from an expert also, but have some experience with my own
RAID 5 array using mdadm.

It looks like one drive is giving you an issue when you are trying to
force the assembly:

mack ~ # mdadm -A --force /dev/md0 /dev/sd{a,b,c,d,e}3
mdadm: superblock on /dev/sdd3 doesn't match others - assembly aborted

Have you tried assembling the array degraded to see if you can force
it without sdd3?  I don't know if this is the best solution, but if
that works, you can then readd sdd3 and have the array rebuild.

Re: Raid 5 Recovery

[Erik Boettcher]

---------- Forwarded message ----------
From: Erik Boettcher <tehfoo@gmail.com>
Date: Sun, Dec 28, 2008 at 1:02 PM
Subject: Re: Raid 5 Recovery
To: Dylan Distasio <interzone@gmail.com>


On Sun, Dec 28, 2008 at 10:53 AM, Dylan Distasio <interzone@gmail.com> wrote:
> Have you tried assembling the array degraded to see if you can force
> it without sdd3?  I don't know if this is the best solution, but if
> that works, you can then readd sdd3 and have the array rebuild.

I tried forcing the assembly without sdd3, and it complains about sde3!

I find it interesting that before the first command sdc3 failed but
now it's not the fault point....

Re: Raid 5 Recovery

[Erik Boettcher]

---------- Forwarded message ----------
From: Erik Boettcher <tehfoo@gmail.com>
Date: Sun, Dec 28, 2008 at 1:04 PM
Subject: Re: Raid 5 Recovery
To: Dylan Distasio <interzone@gmail.com>


On Sun, Dec 28, 2008 at 1:02 PM, Erik Boettcher <tehfoo@gmail.com> wrote:
> On Sun, Dec 28, 2008 at 10:53 AM, Dylan Distasio <interzone@gmail.com> wrote:
>> Have you tried assembling the array degraded to see if you can force
>> it without sdd3?  I don't know if this is the best solution, but if
>> that works, you can then readd sdd3 and have the array rebuild.
>
> I tried forcing the assembly without sdd3, and it complains about sde3!
>
> I find it interesting that before the first command sdc3 failed but
> now it's not the fault point....
>

I guess I should specify how it complains: the exact same error
message but sde3 instead.

mack ~ # mdadm -A --force --no-degraded /dev/md0 /dev/sd{a,b,c,e}3
mdadm: superblock on /dev/sde3 doesn't match others - assembly aborted

eek, and I tried with the first 3 hdds and got this:
mack ~ # mdadm -A --force --no-degraded /dev/md0 /dev/sd{a,b,c}3
mdadm: clearing FAULTY flag for device 2 in /dev/md0 for /dev/sdc3
mdadm: clearing FAULTY flag for device 1 in /dev/md0 for /dev/sdb3
mdadm: clearing FAULTY flag for device 0 in /dev/md0 for /dev/sda3
mdadm: failed to add /dev/sdb3 to /dev/md0: Invalid argument
mdadm: failed to add /dev/sda3 to /dev/md0: Invalid argument
mdadm: failed to add /dev/sdc3 to /dev/md0: Invalid argument
mdadm: /dev/md0 assembled from 0 drives (out of 5), but not started.

Re: Raid 5 Recovery

[NeilBrown]

On Mon, December 29, 2008 5:13 am, Erik Boettcher wrote:
> ---------- Forwarded message ----------
> From: Erik Boettcher <tehfoo@gmail.com>
> Date: Sun, Dec 28, 2008 at 1:04 PM
> Subject: Re: Raid 5 Recovery
> To: Dylan Distasio <interzone@gmail.com>
>
>
> On Sun, Dec 28, 2008 at 1:02 PM, Erik Boettcher <tehfoo@gmail.com> wrote:
>> On Sun, Dec 28, 2008 at 10:53 AM, Dylan Distasio <interzone@gmail.com>
>> wrote:
>>> Have you tried assembling the array degraded to see if you can force
>>> it without sdd3?  I don't know if this is the best solution, but if
>>> that works, you can then readd sdd3 and have the array rebuild.
>>
>> I tried forcing the assembly without sdd3, and it complains about sde3!
>>
>> I find it interesting that before the first command sdc3 failed but
>> now it's not the fault point....
>>
>
> I guess I should specify how it complains: the exact same error
> message but sde3 instead.

Yes, specifics are good.  I'm afraid I need a few more though.
  mdadm -Esvvv
and
  dmesg | tail -20

after the failed --assemble attempt would be a good start.

NeilBrown


>
> mack ~ # mdadm -A --force --no-degraded /dev/md0 /dev/sd{a,b,c,e}3
> mdadm: superblock on /dev/sde3 doesn't match others - assembly aborted
>
> eek, and I tried with the first 3 hdds and got this:
> mack ~ # mdadm -A --force --no-degraded /dev/md0 /dev/sd{a,b,c}3
> mdadm: clearing FAULTY flag for device 2 in /dev/md0 for /dev/sdc3
> mdadm: clearing FAULTY flag for device 1 in /dev/md0 for /dev/sdb3
> mdadm: clearing FAULTY flag for device 0 in /dev/md0 for /dev/sda3
> mdadm: failed to add /dev/sdb3 to /dev/md0: Invalid argument
> mdadm: failed to add /dev/sda3 to /dev/md0: Invalid argument
> mdadm: failed to add /dev/sdc3 to /dev/md0: Invalid argument
> mdadm: /dev/md0 assembled from 0 drives (out of 5), but not started.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Raid 5 Recovery

[Justin Piszcz]

On Sun, 28 Dec 2008, Shane W wrote:

> Hi,
>
> On Sun, Dec 28, 2008 at 01:13:15PM -0500, Erik Boettcher wrote:
>> I guess I should specify how it complains: the exact same error
>> message but sde3 instead.
>
> As others have said, definitely wait for a more
> knowledgable person to post here but if I am working on a
> machine and accidentally jar the SATA cables, mdadm
> sometimes kicks two drives from my raid5 causing it to
> display similar problems.  What works for me is a mdadm -E
> /dev/sdnx where the dev is one of the drives it kicked as
> that represents the "good" state of the array.  Then
> /recreate with one drive missing:
> mdadm --create /dev/mdx --assume-clean --level 5 /dev/sda3
> //dev/sdb3 missing /dev/sdc3 etc. with the device order
> //being the same as that listed in mdadm -E component
> //device list. and missing for the first device which was
> //kicked.  The number of devices, spares, chunk size etc
> //has to match the array of course, then verify the
> //filesystem readonly and once you're sure the order is
> //right and the FS is good, re-add the last device and let
> //the resync do its thing.
>
> I am curious as to whether there is another way to recover
> from the multiple drive nonfailure condition though because
> recreating can raise the heart rate a little.

I was also going to suggest this; however this should be marked as the 
option of last resort, I did this with an array once and used the exact same 
options and much of the FS was corrupted.  If all other options are 
exhausted and you have confirmed that, then it is your last hope I 
believe.

Justin.

Re: Raid 5 Recovery

[Erik Boettcher]

oops, forgot to send it here...


---------- Forwarded message ----------
From: Erik Boettcher <tehfoo@gmail.com>
Date: Sun, Dec 28, 2008 at 4:59 PM
Subject: Re: Raid 5 Recovery
To: NeilBrown <neilb@suse.de>


>
> Yes, specifics are good.  I'm afraid I need a few more though.
>  mdadm -Esvvv

livecd ~ # mdadm -Esvvv
/dev/sde3:
         Magic : a92b4efc
       Version : 00.90.00
          UUID : b2cca5f0:59a04cfc:a3bbe707:108124a2
 Creation Time : Thu Jan 18 02:29:09 2007
    Raid Level : raid5
 Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
    Array Size : 780372992 (744.22 GiB 799.10 GB)
  Raid Devices : 5
 Total Devices : 5
Preferred Minor : 0

   Update Time : Tue Dec  9 02:11:56 2008
         State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
 Spare Devices : 0
      Checksum : 46384ff - correct
        Events : 0.4508

        Layout : left-asymmetric
    Chunk Size : 32K

     Number   Major   Minor   RaidDevice State
this     2       8       67        2      active sync   /dev/sde3

  0     0       8       35        0      active sync   /dev/sdc3
  1     1       8       51        1      active sync   /dev/sdd3
  2     2       8       67        2      active sync   /dev/sde3
  3     3       8       19        3      active sync   /dev/sdb3
  4     4       8        3        4      active sync   /dev/sda3
mdadm: No md superblock detected on /dev/sde2.
mdadm: No md superblock detected on /dev/sde1.
mdadm: No md superblock detected on /dev/sde.
/dev/sdd3:
         Magic : a92b4efc
       Version : 00.90.00
          UUID : b2cca5f0:59a04cfc:a3bbe707:108124a2
 Creation Time : Thu Jan 18 02:29:09 2007
    Raid Level : raid5
 Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
    Array Size : 780372992 (744.22 GiB 799.10 GB)
  Raid Devices : 5
 Total Devices : 5
Preferred Minor : 0

   Update Time : Tue Dec  9 02:11:56 2008
         State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
 Spare Devices : 0
      Checksum : 46384ed - correct
        Events : 0.4508

        Layout : left-asymmetric
    Chunk Size : 32K

     Number   Major   Minor   RaidDevice State
this     1       8       51        1      active sync   /dev/sdd3

  0     0       8       35        0      active sync   /dev/sdc3
  1     1       8       51        1      active sync   /dev/sdd3
  2     2       8       67        2      active sync   /dev/sde3
  3     3       8       19        3      active sync   /dev/sdb3
  4     4       8        3        4      active sync   /dev/sda3
mdadm: No md superblock detected on /dev/sdd2.
mdadm: No md superblock detected on /dev/sdd1.
mdadm: No md superblock detected on /dev/sdd.
/dev/sdc3:
         Magic : a92b4efc
       Version : 00.88.00
          UUID : b2cca5f0
 Creation Time : Thu Jan 18 02:29:09 2007
    Raid Level : raid5
 Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
    Array Size : 780372992 (744.22 GiB 799.10 GB)
  Raid Devices : 5
 Total Devices : 5
Preferred Minor : 0

   Update Time : Tue Dec  9 02:11:56 2008
         State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
 Spare Devices : 0
      Checksum : 46384b9 - correct
        Events : 0.4508

        Layout : left-asymmetric
    Chunk Size : 32K

     Number   Major   Minor   RaidDevice State
this     0       8       33        0      sync   /dev/sdc1

  0     0       8       33        0      active sync   /dev/sdc1
  1     1       8       49        1      active sync   /dev/sdd1
  2     0       8       65        0      sync   /dev/sde1
  3     1       8       17        1      sync   /dev/sdb1
  4     4       8        1        4      active sync   /dev/sda1
mdadm: No md superblock detected on /dev/sdc2.
mdadm: No md superblock detected on /dev/sdc1.
mdadm: No md superblock detected on /dev/sdc.
/dev/sdb3:
         Magic : a92b4efc
       Version : 00.88.00
          UUID : b2cca5f0
 Creation Time : Thu Jan 18 02:29:09 2007
    Raid Level : raid5
 Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
    Array Size : 780372992 (744.22 GiB 799.10 GB)
  Raid Devices : 5
 Total Devices : 5
Preferred Minor : 0

   Update Time : Tue Dec  9 02:11:56 2008
         State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
 Spare Devices : 0
      Checksum : 46384d1 - expected 46384a5
        Events : 0.4508

        Layout : left-asymmetric
    Chunk Size : 32K

     Number   Major   Minor   RaidDevice State
this     1       8       17        1      sync   /dev/sdb1

  0     0       8       33        0      sync   /dev/sdc1
  1     1       8       49        1      sync   /dev/sdd1
  2     0       8       65        0      sync   /dev/sde1
  3     1       8       17        1      sync   /dev/sdb1
  4     4       8        1        4      sync   /dev/sda1
mdadm: No md superblock detected on /dev/sdb2.
mdadm: No md superblock detected on /dev/sdb1.
mdadm: No md superblock detected on /dev/sdb.
/dev/sda3:
         Magic : a92b4efc
       Version : 00.88.00
          UUID : b2cca5f0
 Creation Time : Thu Jan 18 02:29:09 2007
    Raid Level : raid5
 Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
    Array Size : 780372992 (744.22 GiB 799.10 GB)
  Raid Devices : 5
 Total Devices : 5
Preferred Minor : 0

   Update Time : Tue Dec  9 02:11:56 2008
         State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
 Spare Devices : 0
      Checksum : 46384c1 - expected 463849b
        Events : 0.4508

        Layout : left-asymmetric
    Chunk Size : 32K

     Number   Major   Minor   RaidDevice State
this     4       8        1        4      sync   /dev/sda1

  0     0       8       33        0      sync   /dev/sdc1
  1     1       8       49        1      sync   /dev/sdd1
  2     0       8       65        0      sync   /dev/sde1
  3     1       8       17        1      sync   /dev/sdb1
  4     4       8        1        4      sync   /dev/sda1
mdadm: No md superblock detected on /dev/sda2.
mdadm: No md superblock detected on /dev/sda1.
mdadm: No md superblock detected on /dev/sda.
mdadm: No md superblock detected on /dev/loop0.

> and
>  dmesg | tail -20
dmesg only reports what's shown in the first message and this after
each assemble attempt:
md: md0 stopped.

Re: Raid 5 Recovery

[Erik Boettcher]

Is there a way to recover that metadata and edit only the parts that
need to be fixed?

On Sun, Dec 28, 2008 at 6:53 PM, Erik Boettcher <tehfoo@gmail.com> wrote:
> oops, forgot to send it here...
>
>
> ---------- Forwarded message ----------
> From: Erik Boettcher <tehfoo@gmail.com>
> Date: Sun, Dec 28, 2008 at 4:59 PM
> Subject: Re: Raid 5 Recovery
> To: NeilBrown <neilb@suse.de>
>
>
>>
>> Yes, specifics are good.  I'm afraid I need a few more though.
>>  mdadm -Esvvv
>
> livecd ~ # mdadm -Esvvv
> /dev/sde3:
>         Magic : a92b4efc
>       Version : 00.90.00
>          UUID : b2cca5f0:59a04cfc:a3bbe707:108124a2
>  Creation Time : Thu Jan 18 02:29:09 2007
>    Raid Level : raid5
>  Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
>    Array Size : 780372992 (744.22 GiB 799.10 GB)
>  Raid Devices : 5
>  Total Devices : 5
> Preferred Minor : 0
>
>   Update Time : Tue Dec  9 02:11:56 2008
>         State : clean
>  Active Devices : 5
> Working Devices : 5
>  Failed Devices : 0
>  Spare Devices : 0
>      Checksum : 46384ff - correct
>        Events : 0.4508
>
>        Layout : left-asymmetric
>    Chunk Size : 32K
>
>     Number   Major   Minor   RaidDevice State
> this     2       8       67        2      active sync   /dev/sde3
>
>  0     0       8       35        0      active sync   /dev/sdc3
>  1     1       8       51        1      active sync   /dev/sdd3
>  2     2       8       67        2      active sync   /dev/sde3
>  3     3       8       19        3      active sync   /dev/sdb3
>  4     4       8        3        4      active sync   /dev/sda3
> mdadm: No md superblock detected on /dev/sde2.
> mdadm: No md superblock detected on /dev/sde1.
> mdadm: No md superblock detected on /dev/sde.
> /dev/sdd3:
>         Magic : a92b4efc
>       Version : 00.90.00
>          UUID : b2cca5f0:59a04cfc:a3bbe707:108124a2
>  Creation Time : Thu Jan 18 02:29:09 2007
>    Raid Level : raid5
>  Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
>    Array Size : 780372992 (744.22 GiB 799.10 GB)
>  Raid Devices : 5
>  Total Devices : 5
> Preferred Minor : 0
>
>   Update Time : Tue Dec  9 02:11:56 2008
>         State : clean
>  Active Devices : 5
> Working Devices : 5
>  Failed Devices : 0
>  Spare Devices : 0
>      Checksum : 46384ed - correct
>        Events : 0.4508
>
>        Layout : left-asymmetric
>    Chunk Size : 32K
>
>     Number   Major   Minor   RaidDevice State
> this     1       8       51        1      active sync   /dev/sdd3
>
>  0     0       8       35        0      active sync   /dev/sdc3
>  1     1       8       51        1      active sync   /dev/sdd3
>  2     2       8       67        2      active sync   /dev/sde3
>  3     3       8       19        3      active sync   /dev/sdb3
>  4     4       8        3        4      active sync   /dev/sda3
> mdadm: No md superblock detected on /dev/sdd2.
> mdadm: No md superblock detected on /dev/sdd1.
> mdadm: No md superblock detected on /dev/sdd.
> /dev/sdc3:
>         Magic : a92b4efc
>       Version : 00.88.00
>          UUID : b2cca5f0
>  Creation Time : Thu Jan 18 02:29:09 2007
>    Raid Level : raid5
>  Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
>    Array Size : 780372992 (744.22 GiB 799.10 GB)
>  Raid Devices : 5
>  Total Devices : 5
> Preferred Minor : 0
>
>   Update Time : Tue Dec  9 02:11:56 2008
>         State : clean
>  Active Devices : 5
> Working Devices : 5
>  Failed Devices : 0
>  Spare Devices : 0
>      Checksum : 46384b9 - correct
>        Events : 0.4508
>
>        Layout : left-asymmetric
>    Chunk Size : 32K
>
>     Number   Major   Minor   RaidDevice State
> this     0       8       33        0      sync   /dev/sdc1
>
>  0     0       8       33        0      active sync   /dev/sdc1
>  1     1       8       49        1      active sync   /dev/sdd1
>  2     0       8       65        0      sync   /dev/sde1
>  3     1       8       17        1      sync   /dev/sdb1
>  4     4       8        1        4      active sync   /dev/sda1
> mdadm: No md superblock detected on /dev/sdc2.
> mdadm: No md superblock detected on /dev/sdc1.
> mdadm: No md superblock detected on /dev/sdc.
> /dev/sdb3:
>         Magic : a92b4efc
>       Version : 00.88.00
>          UUID : b2cca5f0
>  Creation Time : Thu Jan 18 02:29:09 2007
>    Raid Level : raid5
>  Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
>    Array Size : 780372992 (744.22 GiB 799.10 GB)
>  Raid Devices : 5
>  Total Devices : 5
> Preferred Minor : 0
>
>   Update Time : Tue Dec  9 02:11:56 2008
>         State : clean
>  Active Devices : 5
> Working Devices : 5
>  Failed Devices : 0
>  Spare Devices : 0
>      Checksum : 46384d1 - expected 46384a5
>        Events : 0.4508
>
>        Layout : left-asymmetric
>    Chunk Size : 32K
>
>     Number   Major   Minor   RaidDevice State
> this     1       8       17        1      sync   /dev/sdb1
>
>  0     0       8       33        0      sync   /dev/sdc1
>  1     1       8       49        1      sync   /dev/sdd1
>  2     0       8       65        0      sync   /dev/sde1
>  3     1       8       17        1      sync   /dev/sdb1
>  4     4       8        1        4      sync   /dev/sda1
> mdadm: No md superblock detected on /dev/sdb2.
> mdadm: No md superblock detected on /dev/sdb1.
> mdadm: No md superblock detected on /dev/sdb.
> /dev/sda3:
>         Magic : a92b4efc
>       Version : 00.88.00
>          UUID : b2cca5f0
>  Creation Time : Thu Jan 18 02:29:09 2007
>    Raid Level : raid5
>  Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
>    Array Size : 780372992 (744.22 GiB 799.10 GB)
>  Raid Devices : 5
>  Total Devices : 5
> Preferred Minor : 0
>
>   Update Time : Tue Dec  9 02:11:56 2008
>         State : clean
>  Active Devices : 5
> Working Devices : 5
>  Failed Devices : 0
>  Spare Devices : 0
>      Checksum : 46384c1 - expected 463849b
>        Events : 0.4508
>
>        Layout : left-asymmetric
>    Chunk Size : 32K
>
>     Number   Major   Minor   RaidDevice State
> this     4       8        1        4      sync   /dev/sda1
>
>  0     0       8       33        0      sync   /dev/sdc1
>  1     1       8       49        1      sync   /dev/sdd1
>  2     0       8       65        0      sync   /dev/sde1
>  3     1       8       17        1      sync   /dev/sdb1
>  4     4       8        1        4      sync   /dev/sda1
> mdadm: No md superblock detected on /dev/sda2.
> mdadm: No md superblock detected on /dev/sda1.
> mdadm: No md superblock detected on /dev/sda.
> mdadm: No md superblock detected on /dev/loop0.
>
>> and
>>  dmesg | tail -20
> dmesg only reports what's shown in the first message and this after
> each assemble attempt:
> md: md0 stopped.
>

Re: Raid 5 Recovery

[Erik Boettcher]

[-- Attachment #1: Type: text/plain, Size: 844 bytes --]

On Sun, Dec 28, 2008 at 10:15 PM, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> On Sun, 28 Dec 2008, Shane W wrote:
>> I am curious as to whether there is another way to recover
>> from the multiple drive nonfailure condition though because
>> recreating can raise the heart rate a little.
>
> I was also going to suggest this; however this should be marked as the
> option of last resort, I did this with an array once and used the exact same
> options and much of the FS was corrupted.  If all other options are
> exhausted and you have confirmed that, then it is your last hope I believe.
>

Indeed recreating seems like my only option now.  I'd like to ensure
the right creation order, so I'd like to ask for some help.  Attached
is the output of 'mdadm -Evvvv /dev/sd{a,b,c,d,e}3'  Any help
restoring this array is greatly appreciated!

[-- Attachment #2: mdadmEout --]
[-- Type: application/octet-stream, Size: 5132 bytes --]

/dev/sda3:
          Magic : a92b4efc
        Version : 00.88.00
           UUID : b2cca5f0
  Creation Time : Thu Jan 18 02:29:09 2007
     Raid Level : raid5
  Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
     Array Size : 780372992 (744.22 GiB 799.10 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 0

    Update Time : Tue Dec  9 02:11:56 2008
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 46384c1 - expected 463849b
         Events : 0.4508

         Layout : left-asymmetric
     Chunk Size : 32K

      Number   Major   Minor   RaidDevice State
this     4       8        1        4      sync   /dev/sda1

   0     0       8       33        0      sync   /dev/sdc1
   1     1       8       49        1      sync   /dev/sdd1
   2     0       8       65        0      sync   /dev/sde1
   3     1       8       17        1      sync   /dev/sdb1
   4     4       8        1        4      sync   /dev/sda1
/dev/sdb3:
          Magic : a92b4efc
        Version : 00.88.00
           UUID : b2cca5f0
  Creation Time : Thu Jan 18 02:29:09 2007
     Raid Level : raid5
  Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
     Array Size : 780372992 (744.22 GiB 799.10 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 0

    Update Time : Tue Dec  9 02:11:56 2008
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 46384d1 - expected 46384a5
         Events : 0.4508

         Layout : left-asymmetric
     Chunk Size : 32K

      Number   Major   Minor   RaidDevice State
this     1       8       17        1      sync   /dev/sdb1

   0     0       8       33        0      sync   /dev/sdc1
   1     1       8       49        1      sync   /dev/sdd1
   2     0       8       65        0      sync   /dev/sde1
   3     1       8       17        1      sync   /dev/sdb1
   4     4       8        1        4      sync   /dev/sda1
/dev/sdc3:
          Magic : a92b4efc
        Version : 00.88.00
           UUID : b2cca5f0
  Creation Time : Thu Jan 18 02:29:09 2007
     Raid Level : raid5
  Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
     Array Size : 780372992 (744.22 GiB 799.10 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 0

    Update Time : Tue Dec  9 02:11:56 2008
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 46384b9 - correct
         Events : 0.4508

         Layout : left-asymmetric
     Chunk Size : 32K

      Number   Major   Minor   RaidDevice State
this     0       8       33        0      sync   /dev/sdc1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1
   2     0       8       65        0      sync   /dev/sde1
   3     1       8       17        1      sync   /dev/sdb1
   4     4       8        1        4      active sync   /dev/sda1
/dev/sdd3:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : b2cca5f0:59a04cfc:a3bbe707:108124a2
  Creation Time : Thu Jan 18 02:29:09 2007
     Raid Level : raid5
  Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
     Array Size : 780372992 (744.22 GiB 799.10 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 0

    Update Time : Tue Dec  9 02:11:56 2008
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 46384ed - correct
         Events : 0.4508

         Layout : left-asymmetric
     Chunk Size : 32K

      Number   Major   Minor   RaidDevice State
this     1       8       51        1      active sync   /dev/sdd3

   0     0       8       35        0      active sync   /dev/sdc3
   1     1       8       51        1      active sync   /dev/sdd3
   2     2       8       67        2      active sync   /dev/sde3
   3     3       8       19        3      active sync   /dev/sdb3
   4     4       8        3        4      active sync   /dev/sda3
/dev/sde3:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : b2cca5f0:59a04cfc:a3bbe707:108124a2
  Creation Time : Thu Jan 18 02:29:09 2007
     Raid Level : raid5
  Used Dev Size : 195093248 (186.06 GiB 199.78 GB)
     Array Size : 780372992 (744.22 GiB 799.10 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 0

    Update Time : Tue Dec  9 02:11:56 2008
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 46384ff - correct
         Events : 0.4508

         Layout : left-asymmetric
     Chunk Size : 32K

      Number   Major   Minor   RaidDevice State
this     2       8       67        2      active sync   /dev/sde3

   0     0       8       35        0      active sync   /dev/sdc3
   1     1       8       51        1      active sync   /dev/sdd3
   2     2       8       67        2      active sync   /dev/sde3
   3     3       8       19        3      active sync   /dev/sdb3
   4     4       8        3        4      active sync   /dev/sda3

Re: Raid 5 Recovery

[Max Amanshauser]

On Mar 22, 2009, at 4:15 AM, Erik Boettcher wrote:

> Indeed recreating seems like my only option now.  I'd like to ensure
> the right creation order, so I'd like to ask for some help.  Attached
> is the output of 'mdadm -Evvvv /dev/sd{a,b,c,d,e}3'  Any help
> restoring this array is greatly appreciated!
> <mdadmEout>



I had the same problem recently and solved it relatively pain-free.


Basically what I did was:

*) Find out where the old superblocks are located and backup them with  
dd, just in case. Ideally you should backup all of the disks entirely  
(if they have errors, use dd_rescue and do the following on healthy  
disks).

*) Recreate with mdadm and the same options that were used when  
creating the raid in the first place, with one device declared missing.
"mdadm --create /dev/md0 -e 0.90 -c 32 -p left-asymmetric -l 5 -n 5 / 
dev/sdc3 /dev/sdd3 /dev/sde3 /dev/sdb3 missing" sounds about right in  
your case. I'd find out what the difference between 0.88 and 0.90  
superblocks is, though. If you have devices with errors you should  
declare the most broken one missing, of course. And make absolutely  
sure that the options are correct or you will overwrite innocent data.  
The device order is not so important, since you can change it in  
subsequent recreate attempts without breaking anything.

*) Try to read-only mount the array, and/or use dumpe2fs (if you have  
ext2/3). If it works, hurray. If not, recreate again with a different  
device order. Ultimately you will probably have to e2fsck to correct  
the damage.


I got the correct device order on the second attempt and lost only  
150kB due to hard disk errors. Guess I was lucky.

-- 
Good luck,
Max.

Re: Raid 5 Recovery

[Erik Boettcher]

On Sun, Mar 22, 2009 at 3:15 AM, Max Amanshauser
<linux-raid@lambdalifting.org> wrote:
>
> On Mar 22, 2009, at 4:15 AM, Erik Boettcher wrote:
>
>> Indeed recreating seems like my only option now.  I'd like to ensure
>> the right creation order, so I'd like to ask for some help.  Attached
>> is the output of 'mdadm -Evvvv /dev/sd{a,b,c,d,e}3'  Any help
>> restoring this array is greatly appreciated!
>> <mdadmEout>
>
>
>
> I had the same problem recently and solved it relatively pain-free.
>
>
> Basically what I did was:
>
> *) Find out where the old superblocks are located and backup them with dd,
> just in case. Ideally you should backup all of the disks entirely (if they
> have errors, use dd_rescue and do the following on healthy disks).
Hmm, I can't back up the disks but I'm trying to get the superblocks.
I can't find where they're located to copy them!

>
> *) Recreate with mdadm and the same options that were used when creating the
> raid in the first place, with one device declared missing.
> "mdadm --create /dev/md0 -e 0.90 -c 32 -p left-asymmetric -l 5 -n 5
> /dev/sdc3 /dev/sdd3 /dev/sde3 /dev/sdb3 missing" sounds about right in your
> case. I'd find out what the difference between 0.88 and 0.90 superblocks is,
> though. If you have devices with errors you should declare the most broken
> one missing, of course. And make absolutely sure that the options are
> correct or you will overwrite innocent data. The device order is not so
> important, since you can change it in subsequent recreate attempts without
> breaking anything.
Is the order that mdadm --query provides more correct? I guess I'll find out....

>
> *) Try to read-only mount the array, and/or use dumpe2fs (if you have
> ext2/3). If it works, hurray. If not, recreate again with a different device
> order. Ultimately you will probably have to e2fsck to correct the damage.
LLVM is being used here, so I'll have to make sure vgscan etc don't write...

>
>
> I got the correct device order on the second attempt and lost only 150kB due
> to hard disk errors. Guess I was lucky.
>
> --
> Good luck,
Thanks, I need it.
> Max.

Erik
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid 5 Recovery

[Erik Boettcher]

Ok so I found the superblocks, backed them up, then created the array
(it was sdc,sdd,sde,sda,missing -- luckily I got that first time
around :).  I checked out the first few sectors of /dev/md0 and it
APPEARS to be LLVM, as shown below.  Now before I do anything that
writes the disks any more, I want to be sure that if I'm seeing this
I'm golden.  What do I need to do to bring the last disk back into the
array?

00000210  82 d3 36 66 20 00 00 00  4c 56 4d 32 20 30 30 31  |..6f ...LVM2 001|
00000220  75 67 35 77 55 43 59 6e  34 57 71 39 49 38 53 35  |ug5wUCYn4Wq9I8S5|
00000230  41 31 45 6d 69 79 71 32  51 78 68 42 39 63 32 78  |A1Emiyq2QxhB9c2x|

Many thanks for all of your help!