PLEASE HELP ... raid5 array degraded ...

PLEASE HELP ... raid5 array degraded ...

[Andrew Skolarz]

as the topic says i was running a raid5 array.  i was using this
because i assumed it wouldn't die .. even if i lost a drive.  well, i
guess i was wrong :(

a little background .. i have 2 raid5 arrays.  one is compromised of 4
200gb hd's (this crashed as of last night) and one is 4 300gb's.  last
night, i was accessing a file on the 2x200 while my server decided to
crash.  When i booted up today, the boot up sequence started yelling
at me that it couldn't load the reiserfs cause of a bad superblock.
upon further investigation i determined it wasn't the a bad superblock
in the reiserfs .. but rather one of the md superblocks on my hd's was
corrupt.

myth ~ # dmesg | grep md
md: raid0 personality registered as nr 2
md: raid5 personality registered as nr 4
md: md driver 0.90.2 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 3.38
md: Autodetecting RAID arrays.
md: invalid superblock checksum on hdl1
md: hdl1 has invalid sb, not importing!
md: autorun ...
md: considering hdk1 ...
md:  adding hdk1 ...
md:  adding hdj1 ...
md:  adding hdi1 ...
md: created md0
md: bind<hdi1>
md: bind<hdj1>
md: bind<hdk1>
md: running: <hdk1><hdj1><hdi1>
md: md0: raid array is not clean -- starting background reconstruction
raid5: cannot start dirty degraded array for md0
raid5: failed to run raid set md0
md: pers->run() failed ...
md: do_md_run() returned -5
md: md0 stopped.
md: unbind<hdk1>
md: export_rdev(hdk1)
md: unbind<hdj1>
md: export_rdev(hdj1)
md: unbind<hdi1>
md: export_rdev(hdi1)
md: ... autorun DONE.

is the whole array dead ?? is there no way to recover the 450gb of
data i have on there ?

i tried even unplugging /dev/hdl and prayed it would start up with
just 3 drives (like raid5 should) but no luck .. just like here
without that drive, it doesn't want to reconstruct.

PLEASE HELP and tell me that my data is not lost .. i beg you

Re: PLEASE HELP ... raid5 array degraded ...

[Andrew Skolarz]

i performed mdadm -E on the devices in the array

myth ~ # mdadm -E /dev/hdi1
/dev/hdi1:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : 9ab2395c:b39cfe1c:ba195ea0:0505eb9d
  Creation Time : Mon Dec 26 13:41:06 2005
     Raid Level : raid5
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Mon Jul 10 21:43:11 2006
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 4aaf93b4 - correct
         Events : 0.428721

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0      56        1        0      active sync   /dev/hdi1

   0     0      56        1        0      active sync   /dev/hdi1
   1     1      56       65        1      active sync   /dev/hdj1
   2     2      57        1        2      active sync   /dev/hdk1
   3     3      57       65        3      active sync   /dev/hdl1


myth ~ # mdadm -E /dev/hdj1
/dev/hdj1:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : 9ab2395c:b39cfe1c:ba195ea0:0505eb9d
  Creation Time : Mon Dec 26 13:41:06 2005
     Raid Level : raid5
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Mon Jul 10 21:43:11 2006
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 4aaf93f6 - correct
         Events : 0.428721

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1      56       65        1      active sync   /dev/hdj1

   0     0      56        1        0      active sync   /dev/hdi1
   1     1      56       65        1      active sync   /dev/hdj1
   2     2      57        1        2      active sync   /dev/hdk1
   3     3      57       65        3      active sync   /dev/hdl1


myth ~ # mdadm -E /dev/hdk1
/dev/hdk1:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : 9ab2395c:b39cfe1c:ba195ea0:0505eb9d
  Creation Time : Mon Dec 26 13:41:06 2005
     Raid Level : raid5
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Mon Jul 10 21:43:11 2006
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 4aa90908 - correct
         Events : 0.428722

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2      57        1        2      active sync   /dev/hdk1

   0     0      56        1        0      active sync   /dev/hdi1
   1     1      56       65        1      active sync   /dev/hdj1
   2     2      57        1        2      active sync   /dev/hdk1
   3     3      57       65        3      active sync   /dev/hdl1


myth ~ # mdadm -E /dev/hdl1
/dev/hdl1:
          Magic : a92b4efc
        Version : 00.90.02
           UUID : 9ab2395c:b39cfe1c:ba195ea0:0505eb9d
  Creation Time : Mon Dec 26 13:41:06 2005
     Raid Level : raid5
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Mon Jul 10 21:43:11 2006
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 4aa9094a - expected 4aa908c4
         Events : 0.428722

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       0        0        0      spare

   0     0      56        1        0      active sync   /dev/hdi1
   1     1      56       65        1      active sync   /dev/hdj1
   2     2      57        1        2      active sync   /dev/hdk1
   3     3      57       65        3      active sync   /dev/hdl1


it does report as hdk having a bad checksum .. but the other 3 drives
seem okay .. hopefully i can restore my data ... but why isn't it
loading up in a 3 drive config ??

Re: PLEASE HELP ... raid5 array degraded ...

[Andrew Skolarz]

please disregard this email .. after doing more google research i have
re-assembled the array and once again am a true believer of software
raid

BEHOLD THE POWER OF MD

Re: PLEASE HELP ... raid5 array degraded - how fix it?

[Dexter Filmore]

Am Mittwoch, 12. Juli 2006 06:10 schrieben Sie:

Care to enlighten the rest of us what did the trick?

Dex

> please disregard this email .. after doing more google research i have
> re-assembled the array and once again am a true believer of software
> raid
>
> BEHOLD THE POWER OF MD
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C++++ UL++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS+ PE Y++ PGP t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b++(+++) DI+++ D- G++ e* h>++ r* y?
------END GEEK CODE BLOCK------

http://www.stop1984.com
http://www.againsttcpa.com

Re: PLEASE HELP ... raid5 array degraded - how fix it?

[Andrew Skolarz]

i just forced a re-assemble ... and then it fixed it self

mdm -A --force /dev/md0 /dev/hd[ijk]1  (forced re-assemble on the 3 good drives)
mdm -a /dev/md0 /dev/hdl1 (re-added the bad drive and it fixed itself)

Re: PLEASE HELP ... raid5 array degraded ...

[Neil Brown]

On Tuesday July 11, pletopia@gmail.com wrote:
>        Checksum : 4aa9094a - expected 4aa908c4

This is a bit scary.  You have a single-bit error, either in the
checksum or elsewhere in the superblock.

I would recommend at least a memtest86 run.

NeilBrown