RAID5 with 2 drive failure at the same time

[Christoph Nelles <evilazrael@evilazrael.de>]

[-- Attachment #1: Type: text/plain, Size: 2369 bytes --]

Hi,

i hope somebody on this ML can help me.

My RAID5 died last night during a rebuild when two drives failed (looks
like a sata_mv problem). The RAID5 was rebuilding because one of the two
drives failed before and after running badblocks for 2 days, i re-added
it to the RAID.

The used drives are from /dev/sdb1 to /dev/sdj1 (9 Drives, RAID5), the
failed drives are sdj1 and sdg1
The current situation is that I cannot start the RAID. I wanted to try
readding on of the the drives, so removed it beforehand, making it a
spare :\ The layout is as follows:

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       0        0        1      removed
       2       8      113        2      active sync   /dev/sdh1
       3       8       49        3      active sync   /dev/sdd1
       4       8      129        4      active sync   /dev/sdi1
       5       0        0        5      removed
       6       8       17        6      active sync   /dev/sdb1
       7       8       81        7      active sync   /dev/sdf1
       8       8       65        8      active sync   /dev/sde1

Re-adding fails with a simple message:
# mdadm -v /dev/md0 --re-add /dev/sdg1
mdadm: --re-add for /dev/sdg1 to /dev/md0 is not possible

I tried re-adding both failed drives at the same, with the same result.

When examining the drives, sdj1 has the information from before the crash:
   Device Role : Active device 5
   Array State : AAAAAAAAA ('A' == active, '.' == missing)

sdg1 looks like this
   Device Role : spare
   Array State : A.AAA.AAA ('A' == active, '.' == missing)

The other look like
   Device Role : Active device 6
   Array State : A.AAA.AAA ('A' == active, '.' == missing)

So looks that my repair tries made sdg1 a spare :\ I attached the full
output to this mail.

Is there anyway to restart the RAID from the information contained in
drive sdj1? Perhaps via Incremental Build starting from one drive? Could
that work? If the RAID wouldn't have been rebuilding before the crash, i
would just recreate it with --assume-clean.

Thanks in advance for any help

Regards

Christoph Nelles
-- 
Christoph Nelles

E-Mail    : evilazrael@evilazrael.de
Jabber    : eazrael@evilazrael.net      ICQ       : 78819723

PGP-Key   : ID 0x424FB55B on subkeys.pgp.net
            or http://evilazrael.net/pgp.txt


[-- Attachment #2: mdadm_examine_sdg1.txt --]
[-- Type: text/plain, Size: 849 bytes --]

# mdadm --examine /dev/sdg1
/dev/sdg1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6b21b3ed:d39d5a54:d4939113:77851cb6
           Name : router:0  (local to host router)
  Creation Time : Fri Apr 27 20:25:04 2012
     Raid Level : raid5
   Raid Devices : 9

 Avail Dev Size : 5860529039 (2794.52 GiB 3000.59 GB)
     Array Size : 46884229120 (22356.14 GiB 24004.73 GB)
  Used Dev Size : 5860528640 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : a1b16284:321fcdd0:93993ff5:832eee3a

    Update Time : Thu Jan 31 00:50:44 2013
       Checksum : 2391e873 - correct
         Events : 27697

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : spare
   Array State : A.AAA.AAA ('A' == active, '.' == missing)

[-- Attachment #3: mdadm_examine_sdj1.txt --]
[-- Type: text/plain, Size: 857 bytes --]

mdadm --examine /dev/sdj1
/dev/sdj1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6b21b3ed:d39d5a54:d4939113:77851cb6
           Name : router:0  (local to host router)
  Creation Time : Fri Apr 27 20:25:04 2012
     Raid Level : raid5
   Raid Devices : 9

 Avail Dev Size : 5860529039 (2794.52 GiB 3000.59 GB)
     Array Size : 46884229120 (22356.14 GiB 24004.73 GB)
  Used Dev Size : 5860528640 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 7023df83:d890ce04:fc28652e:094adffe

    Update Time : Thu Jan 31 00:24:56 2013
       Checksum : 542f70be - correct
         Events : 27691

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 5
   Array State : AAAAAAAAA ('A' == active, '.' == missing)

[-- Attachment #4: mdadm_detail.txt --]
[-- Type: text/plain, Size: 1207 bytes --]

 mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Fri Apr 27 20:25:04 2012
     Raid Level : raid5
  Used Dev Size : -1
   Raid Devices : 9
  Total Devices : 7
    Persistence : Superblock is persistent

    Update Time : Thu Jan 31 10:36:28 2013
          State : active, FAILED, Not Started
 Active Devices : 7
Working Devices : 7
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           Name : router:0  (local to host router)
           UUID : 6b21b3ed:d39d5a54:d4939113:77851cb6
         Events : 27699

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       0        0        1      removed
       2       8      113        2      active sync   /dev/sdh1
       3       8       49        3      active sync   /dev/sdd1
       4       8      129        4      active sync   /dev/sdi1
       5       0        0        5      removed
       6       8       17        6      active sync   /dev/sdb1
       7       8       81        7      active sync   /dev/sdf1
       8       8       65        8      active sync   /dev/sde1

[-- Attachment #5: mdadm_examine_sdb1.txt --]
[-- Type: text/plain, Size: 857 bytes --]

/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6b21b3ed:d39d5a54:d4939113:77851cb6
           Name : router:0  (local to host router)
  Creation Time : Fri Apr 27 20:25:04 2012
     Raid Level : raid5
   Raid Devices : 9

 Avail Dev Size : 5860529039 (2794.52 GiB 3000.59 GB)
     Array Size : 46884229120 (22356.14 GiB 24004.73 GB)
  Used Dev Size : 5860528640 (2794.52 GiB 3000.59 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 29c62776:e9c58ce6:1c6e9ab1:046ac411

    Update Time : Thu Jan 31 10:36:28 2013
       Checksum : be473d02 - correct
         Events : 27699

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 6
   Array State : A.AAA.AAA ('A' == active, '.' == missing)