raid1 recoverable after system crash?

["Brian J. Murrell" <brian@interlinx.bc.ca>]

[-- Attachment #1: Type: text/plain, Size: 4193 bytes --]

Hi

I had a system crash.  When it came back up, one of my arrays was
degraded:

# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sdd[0]
      1953514496 blocks [2/1] [U_]
      
md1 : active raid0 sdc[1] sdb[0]
      1953524736 blocks super 1.2 512k chunks
      
unused devices: <none>

md1 is supposed to be a member of md0 but it's not currently:

/dev/md0:
        Version : 0.90
  Creation Time : Mon Jan 26 19:51:38 2009
     Raid Level : raid1
     Array Size : 1953514496 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Apr  7 08:19:25 2016
          State : clean, degraded 
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           UUID : 2f8fc5e0:a0eb646a:2303d005:33f25f21
         Events : 0.5030755

    Number   Major   Minor   RaidDevice State
       0       8       48        0      active sync   /dev/sdd
       1       0        0        1      removed

It doesn't seem to be re-addable:

# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --manage /dev/md0 --re-add /dev/md1
mdadm: --re-add for /dev/md1 to /dev/md0 is not possible

It doesn't seem to be assemble-able:

# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --assemble /dev/md0 /dev/sdd /dev/md1
mdadm: /dev/md0 has been started with 1 drive (out of 2).

even when forced:

# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --assemble --force /dev/md0 /dev/sdd /dev/md1
mdadm: /dev/md0 has been started with 1 drive (out of 2).

Is my only option here to fail/remove /dev/md1 from the array and re-
add it that way or is there a more graceful recovery possible here?

Some additional info:

mdadm --examine /dev/md1
/dev/md1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 2f8fc5e0:a0eb646a:2303d005:33f25f21
  Creation Time : Mon Jan 26 19:51:38 2009
     Raid Level : raid1
  Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
     Array Size : 1953514496 (1863.02 GiB 2000.40 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Thu Apr  7 00:18:47 2016
          State : active
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : e5ddeac5 - correct
         Events : 5030737


      Number   Major   Minor   RaidDevice State
this     1       9        1        1      active sync   /dev/md1
   0     0       8       48        0      active sync   /dev/sdd
   1     1       9        1        1      active sync   /dev/md1
# mdadm --examine /dev/sdd
/dev/sdd:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 2f8fc5e0:a0eb646a:2303d005:33f25f21
  Creation Time : Mon Jan 26 19:51:38 2009
     Raid Level : raid1
  Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
     Array Size : 1953514496 (1863.02 GiB 2000.40 GB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0

    Update Time : Thu Apr  7 08:37:04 2016
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0
       Checksum : e62b232b - correct
         Events : 5030757


      Number   Major   Minor   RaidDevice State
this     0       8       48        0      active sync   /dev/sdd

   0     0       8       48        0      active sync   /dev/sdd
   1     1       0        0        1      faulty removed

Most happy to provide any additional information needed.

Cheers,
b.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]