Need help with my raid5

Need help with my raid5

[Mirko]

Hi,

i wanted to reassemble my Raid5.
Last week i added a new device to my raid5 with 6 devices.
After the reshape, mdadm said, device 7 is failed. I tested the device and 
there are a lot of bad blocks.
So i removed the device from the raid, resized the filesystem and during the
reshape, device #3 was removed from the raid and the reshape was canceled.
                                                                                
Yesterday i removed the 7th device from the computer and now, 
the raid doesnt start.

mdadm --assemble --verbose /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
/dev/sde1/dev/sdf1
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot -1.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 2.
mdadm: device 7 in /dev/md0 has wrong state in superblock, but /dev/sdc1 seems
ok
mdadm:/dev/md0 has an active reshape - checking if critical section needs to be
restored
mdadm: added /dev/sda1 to /dev/md0 as 1
mdadm: added /dev/sdf1 to /dev/md0 as 2
mdadm: no uptodate device for slot 3 of /dev/md0
mdadm: added /dev/sde1 to /dev/md0 as 4
mdadm: added /dev/sdb1 to /dev/md0 as 5
mdadm: no uptodate device for slot 6 of /dev/md0
mdadm: added /dev/sdc1 to /dev/md0 as -1
mdadm: added /dev/sdd1 to /dev/md0 as 0
mdadm: /dev/md0 assembled from 5 drives and 1 spare - not enough to start the
array.

mdadm --run /dev/md0
mdadm: failed to run array /dev/md0: Input/output error

mdadm -D /dev/md0
/dev/md0:

        Version : 1.2
  Creation Time : Fri Feb 22 22:08:12 2013
     Raid Level : raid5
  Used Dev Size : 976630272 (931.39 GiB 1000.07 GB)
   Raid Devices : 6
  Total Devices : 6
    Persistence : Superblock is persistent

    Update Time : Thu Mar 21 15:50:39 2013
          State : active, degraded, Not Started
 Active Devices : 5
 Working Devices : 6
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

  Delta Devices : -1, (7->6)

           Name : server:0  (local to host server)
           UUID : 1db312ca:48cd4f30:e8607949:0c721b94
         Events : 38267

    Number   Major   Minor   RaidDevice State
       0       8       49        0      active sync   /dev/sdd1
       1       8        1        1      active sync   /dev/sda1
       3       8       81        2      active sync   /dev/sdf1
       3       0        0        3      removed
       5       8       65        4      active sync   /dev/sde1
       6       8       17        5      active sync   /dev/sdb1
       7       8       33        -      spare   /dev/sdc1

What can i do to rescue my data?

Thx

Re: Need help with my raid5

[Roger Heflin]

On Fri, Mar 22, 2013 at 2:57 AM, Mirko <mirko76@arcor.de> wrote:

>
>     Number   Major   Minor   RaidDevice State
>        0       8       49        0      active sync   /dev/sdd1
>        1       8        1        1      active sync   /dev/sda1
>        3       8       81        2      active sync   /dev/sdf1
>        3       0        0        3      removed
>        5       8       65        4      active sync   /dev/sde1
>        6       8       17        5      active sync   /dev/sdb1
>        7       8       33        -      spare   /dev/sdc1
>
> What can i do to rescue my data?
>
> Thx
>
> --

According to the above you removed device 3 not device 7....device 3
is missing...device 7 is still there and is a spare.

Put what you though was 7 back...note that properly determining which
device is which is pretty difficult unless you have a commercial grade
wired enclosure from a tier 1 vendor that is actually labeled and
wired correctly (don't always count on that).

I have self wired hot swap enclosures and my method is to read from
the array and see which disk does not light up (confirm all others are
active) and then cancel that read and go back and read from the bad
disk to confirm which device that is, and if I want to be really save
cancel that read make sure that disk stops then read again just to
confirm I know what is what, no matter what you have, no matter how it
is incorrectly labeled/wired this method should always work right.

Re: Need help with my raid5

[Brad Campbell]

On 22/03/13 20:06, Roger Heflin wrote:

> I have self wired hot swap enclosures and my method is to read from
> the array and see which disk does not light up (confirm all others are
> active) and then cancel that read and go back and read from the bad
> disk to confirm which device that is, and if I want to be really save
> cancel that read make sure that disk stops then read again just to
> confirm I know what is what, no matter what you have, no matter how it
> is incorrectly labeled/wired this method should always work right.

Just as a data point. I have a spreadsheet with the serial number of 
each drive tied to the physical drive slot, so I know *exactly* which 
drive to pull when one dies. Use hdparm to give you a list of serial 
numbers to cut and paste into open-office, then just shuffle the cells 
around as you pull each drive to check it's number. Takes no time to 
create when you next have the machine down.

Your method is good too, and I've used that when in a hurry and not 
likely to cause damage if I get it wrong.

Re: Need help with my raid5

[Mirko]

> According to the above you removed device 3 not device 7....device 3
> is missing...device 7 is still there and is a spare.

Thats right. 
 
> Put what you though was 7 back
This device no longer exists.

> ...note that properly determining which
> device is which is pretty difficult unless you have a commercial grade
> wired enclosure from a tier 1 vendor that is actually labeled and
> wired correctly (don't always count on that).

I thought, every device has its fixed device number in the raid:

mdadm -E /dev/sd[a-f]1 | grep Role
   Device Role : Active device 1
   Device Role : Active device 5
   Device Role : spare
   Device Role : Active device 0
   Device Role : Active device 4
   Device Role : Active device 2

Re: Need help with my raid5

[Mirko]

I added the damaged device again to the raid:

/dev/md0:
        Version : 1.2
  Creation Time : Fri Feb 22 22:08:12 2013
     Raid Level : raid5
  Used Dev Size : 976630272 (931.39 GiB 1000.07 GB)
   Raid Devices : 6
  Total Devices : 7
    Persistence : Superblock is persistent

    Update Time : Thu Mar 21 15:50:39 2013
          State : active, degraded, Not Started 
 Active Devices : 5
Working Devices : 7
 Failed Devices : 0
  Spare Devices : 2

         Layout : left-symmetric
     Chunk Size : 512K

  Delta Devices : -1, (7->6)

           Name : server:0  (local to host server)
           UUID : 1db312ca:48cd4f30:e8607949:0c721b94
         Events : 38267

    Number   Major   Minor   RaidDevice State
       0       8       49        0      active sync   /dev/sdd1
       1       8        1        1      active sync   /dev/sda1
       3       8       81        2      active sync   /dev/sdf1
       3       0        0        3      removed
       5       8       65        4      active sync   /dev/sde1
       6       8       17        5      active sync   /dev/sdb1

       7       8       33        -      spare   /dev/sdc1
       8       8      113        -      spare   /dev/sdh1

But now i have two spare drives.

Re: Need help with my raid5

[Mirko]

mdadm --assemble --auto=yes --force --verbose /dev/md0 /dev/sda1 /dev/sdb1
/dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdh1 
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 5.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot -1.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot -1.
mdadm: clearing FAULTY flag for device 6 in /dev/md0 for /dev/sdh1
mdadm: Marking array /dev/md0 as 'clean'
mdadm:/dev/md0 has an active reshape - checking if critical section needs to be
restored
mdadm: added /dev/sda1 to /dev/md0 as 1
mdadm: added /dev/sdf1 to /dev/md0 as 2
mdadm: no uptodate device for slot 3 of /dev/md0
mdadm: added /dev/sde1 to /dev/md0 as 4
mdadm: added /dev/sdb1 to /dev/md0 as 5
mdadm: no uptodate device for slot 6 of /dev/md0
mdadm: added /dev/sdc1 to /dev/md0 as -1
mdadm: added /dev/sdh1 to /dev/md0 as -1
mdadm: added /dev/sdd1 to /dev/md0 as 0
mdadm: /dev/md0 assembled from 5 drives and 2 spares - not enough to start the
array.

Re: Need help with my raid5

[Roger Heflin]

On Fri, Mar 22, 2013 at 7:53 AM, Mirko <mirko76@arcor.de> wrote:
>> According to the above you removed device 3 not device 7....device 3
>> is missing...device 7 is still there and is a spare.
>
> Thats right.
>
>> Put what you though was 7 back
> This device no longer exists.

What do you mean no longer exists?

If so, then you are likely restoring from backups as device 7 failed
and device 3 was otherwise was removed and is now no longer viable.
You may be able to do add a -f on the mdadm and force it
online...typically it will force a previously declared missing/spare
device to be re-evaluated and reused if it is good enough or failed
for some reasons.

>
>> ...note that properly determining which
>> device is which is pretty difficult unless you have a commercial grade
>> wired enclosure from a tier 1 vendor that is actually labeled and
>> wired correctly (don't always count on that).
>
> I thought, every device has its fixed device number in the raid:
>
> mdadm -E /dev/sd[a-f]1 | grep Role
>    Device Role : Active device 1
>    Device Role : Active device 5
>    Device Role : spare
>    Device Role : Active device 0
>    Device Role : Active device 4
>    Device Role : Active device 2
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html