RAID6 growing interrupted, array won't assemble or resume growing

RAID6 growing interrupted, array won't assemble or resume growing

[Nic Wolfe]

First a little bit of background about my setup and how I got into this state:

I'm running an older version of ubuntu with a 2.6.24.5 kernel and
mdadm 2.6.3. I had a 5x2TB raid6 array which I attempted to grow to a
6x2TB array. While it was growing I had some hardware problems and the
disks in the array sporadically connected/disconnected. This put the
array in a bad state.

After fixing my hardware issues and getting the PC back up I had a
problem where after booting mdadm would consume all my RAM trying to
assemble my array (oom_killer started killing indiscriminately and I
couldn't get on the PC to shut it down, had to power cycle it). I
added some more memory (from 2GB to 4GB) and mdadm now only takes up
about 70% before it exits with no results that I can tell. Below are
the processes which run when I boot:

root      3052  0.0  0.0   1704   468 ?        S<   23:23   0:00
/lib/udev/watershed /sbin/mdadm --assemble --scan --no-degraded
root      3053  0.0  0.0   1704   460 ?        S<   23:23   0:00
/lib/udev/watershed /sbin/mdadm --assemble --scan --no-degraded
root      3054  0.0  0.0   1704   460 ?        S<   23:23   0:00
/lib/udev/watershed /sbin/mdadm --assemble --scan --no-degraded
root      3055  0.0  0.0   1704   460 ?        S<   23:23   0:00
/lib/udev/watershed /sbin/mdadm --assemble --scan --no-degraded
root      3056  0.0  0.0   1704   464 ?        S<   23:23   0:00
/lib/udev/watershed /sbin/mdadm --assemble --scan --no-degraded
root      6677  0.0  0.0   2084   336 ?        Ss   23:26   0:00
/sbin/mdadm --monitor --pid-file /var/run/mdadm/monitor.pid
--daemonise --scan --syslog
root      7072 42.1 67.1 2768196 2766984 ?     R<   23:42   7:01
/sbin/mdadm --assemble --scan --no-degraded




So anyway now that I have the system stable and all 6 drives hooked up
I would very much like to get the array working again.

I have the following in my mdadm.conf: ARRAY /dev/md1 level=raid6
num-devices=5 UUID=4672ced4:81401dbc:52723fc8:3fe02f5a
(it is currently commented out, note that it didn't get updated after
growing to 6)

Below is the --examine for all 6 drives:

midgetspy@MidgetNAS:~$ sudo mdadm --examine /dev/sda
mdadm: No md superblock detected on /dev/sda.
midgetspy@MidgetNAS:~$ sudo mdadm --examine /dev/sdb
/dev/sdb:
          Magic : a92b4efc
        Version : 00.91.00
           UUID : 4672ced4:81401dbc:52723fc8:3fe02f5a (local to host MidgetNAS)
  Creation Time : Wed Jun  2 21:11:18 2010
     Raid Level : raid6
  Used Dev Size : 1953431488 (1862.94 GiB 2000.31 GB)
     Array Size : 7813725952 (7451.75 GiB 8001.26 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 1

  Reshape pos'n : 665856 (650.36 MiB 681.84 MB)
  Delta Devices : 1 (5->6)

    Update Time : Mon Oct 22 21:06:07 2012
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 2
  Spare Devices : 0
       Checksum : 146b8c4a - correct
         Events : 0.1323352

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8      176        2      active sync

   0     0       0        0        0      removed
   1     1       8      192        1      active sync
   2     2       8      176        2      active sync
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
   5     5      65        0        5      active sync

   midgetspy@MidgetNAS:~$ sudo mdadm --examine /dev/sdc
/dev/sdc:
          Magic : a92b4efc
        Version : 00.91.00
           UUID : 4672ced4:81401dbc:52723fc8:3fe02f5a (local to host MidgetNAS)
  Creation Time : Wed Jun  2 21:11:18 2010
     Raid Level : raid6
  Used Dev Size : 1953431488 (1862.94 GiB 2000.31 GB)
     Array Size : 7813725952 (7451.75 GiB 8001.26 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 1

  Reshape pos'n : 665856 (650.36 MiB 681.84 MB)
  Delta Devices : 1 (5->6)

    Update Time : Mon Oct 22 21:06:07 2012
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 2
  Spare Devices : 0
       Checksum : 146b8c58 - correct
         Events : 0.1323352

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8      192        1      active sync

   0     0       0        0        0      removed
   1     1       8      192        1      active sync
   2     2       8      176        2      active sync
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
   5     5      65        0        5      active sync

   midgetspy@MidgetNAS:~$ sudo mdadm --examine /dev/sdd
/dev/sdd:
          Magic : a92b4efc
        Version : 00.91.00
           UUID : 4672ced4:81401dbc:52723fc8:3fe02f5a (local to host MidgetNAS)
  Creation Time : Wed Jun  2 21:11:18 2010
     Raid Level : raid6
  Used Dev Size : 1953431488 (1862.94 GiB 2000.31 GB)
     Array Size : 7813725952 (7451.75 GiB 8001.26 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 1

  Reshape pos'n : 665856 (650.36 MiB 681.84 MB)
  Delta Devices : 1 (5->6)

    Update Time : Mon Oct 22 21:05:39 2012
          State : clean
 Active Devices : 6
Working Devices : 6
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 146b8c05 - correct
         Events : 0.1323342

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8      208        0      active sync

   0     0       8      208        0      active sync
   1     1       8      192        1      active sync
   2     2       8      176        2      active sync
   3     3       8      224        3      active sync
   4     4       8      240        4      active sync
   5     5      65        0        5      active sync

midgetspy@MidgetNAS:~$ sudo mdadm --examine /dev/sde
/dev/sde:
          Magic : a92b4efc
        Version : 00.91.00
           UUID : 4672ced4:81401dbc:52723fc8:3fe02f5a (local to host MidgetNAS)
  Creation Time : Wed Jun  2 21:11:18 2010
     Raid Level : raid6
  Used Dev Size : 1953431488 (1862.94 GiB 2000.31 GB)
     Array Size : 7813725952 (7451.75 GiB 8001.26 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 1

  Reshape pos'n : 665856 (650.36 MiB 681.84 MB)
  Delta Devices : 1 (5->6)

    Update Time : Mon Oct 22 21:05:58 2012
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 146b8c4b - correct
         Events : 0.1323350

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8      224        3      active sync

   0     0       0        0        0      removed
   1     1       8      192        1      active sync
   2     2       8      176        2      active sync
   3     3       8      224        3      active sync
   4     4       8      240        4      active sync
   5     5      65        0        5      active sync

midgetspy@MidgetNAS:~$ sudo mdadm --examine /dev/sdf
/dev/sdf:
          Magic : a92b4efc
        Version : 00.91.00
           UUID : 4672ced4:81401dbc:52723fc8:3fe02f5a (local to host MidgetNAS)
  Creation Time : Wed Jun  2 21:11:18 2010
     Raid Level : raid6
  Used Dev Size : 1953431488 (1862.94 GiB 2000.31 GB)
     Array Size : 7813725952 (7451.75 GiB 8001.26 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 1

  Reshape pos'n : 665856 (650.36 MiB 681.84 MB)
  Delta Devices : 1 (5->6)

    Update Time : Mon Oct 22 21:05:58 2012
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 146b8c5d - correct
         Events : 0.1323350

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     4       8      240        4      active sync

   0     0       0        0        0      removed
   1     1       8      192        1      active sync
   2     2       8      176        2      active sync
   3     3       8      224        3      active sync
   4     4       8      240        4      active sync
   5     5      65        0        5      active sync


How should I proceed? I'm far enough out of my depth that I'm hesitant
to try anything for fear of causing more damage. Should I update my
mdadm.conf to have num-devices=6 and see if it sorts itself out? Try
to force assemble the 5 drives with superblocks? Create a "new" array
out of them?

Any input would be greatly appreciated.

Thanks,

Nic

Re: RAID6 growing interrupted, array won't assemble or resume growing

[Phil Turmel]

On 06/06/2013 02:41 AM, Nic Wolfe wrote:
> First a little bit of background about my setup and how I got into this state:

Very good report.

> I'm running an older version of ubuntu with a 2.6.24.5 kernel and
> mdadm 2.6.3. I had a 5x2TB raid6 array which I attempted to grow to a
> 6x2TB array. While it was growing I had some hardware problems and the
> disks in the array sporadically connected/disconnected. This put the
> array in a bad state.

The old kernel and mdadm concern me.  Patches go through the mailing
list pretty steadily, both for features and bugs.

> After fixing my hardware issues and getting the PC back up I had a
> problem where after booting mdadm would consume all my RAM trying to
> assemble my array (oom_killer started killing indiscriminately and I
> couldn't get on the PC to shut it down, had to power cycle it). I
> added some more memory (from 2GB to 4GB) and mdadm now only takes up
> about 70% before it exits with no results that I can tell. Below are
> the processes which run when I boot:

This sounds like an udev issue.  Probably not a problem on a stable
system, but you have an intermediate state.

[trim /]

> So anyway now that I have the system stable and all 6 drives hooked up
> I would very much like to get the array working again.
> 
> I have the following in my mdadm.conf: ARRAY /dev/md1 level=raid6
> num-devices=5 UUID=4672ced4:81401dbc:52723fc8:3fe02f5a
> (it is currently commented out, note that it didn't get updated after
> growing to 6)

mdadm is never updated automatically by the vanilla tools.  You get to
do that yourself.  Although you'd be fine to simply remove the level=
and num-devices= clauses.  (Remember to update your initramfs, too.)

> Below is the --examine for all 6 drives:

Yes!  The most important data you could report.

> midgetspy@MidgetNAS:~$ sudo mdadm --examine /dev/sda
> mdadm: No md superblock detected on /dev/sda.
> midgetspy@MidgetNAS:~$ sudo mdadm --examine /dev/sdb
> /dev/sdb:
>           Magic : a92b4efc
>         Version : 00.91.00
                    ^^^^^^^^
This means a normally v0.90 array has a reshape in progress.  That
prevents really old kernels from mistakenly assembling it.

>            UUID : 4672ced4:81401dbc:52723fc8:3fe02f5a (local to host MidgetNAS)
>   Creation Time : Wed Jun  2 21:11:18 2010
>      Raid Level : raid6
>   Used Dev Size : 1953431488 (1862.94 GiB 2000.31 GB)
>      Array Size : 7813725952 (7451.75 GiB 8001.26 GB)
>    Raid Devices : 6
>   Total Devices : 6
> Preferred Minor : 1
> 
>   Reshape pos'n : 665856 (650.36 MiB 681.84 MB)
>   Delta Devices : 1 (5->6)

Your reshape is barely started.  Presumably you specified a --backup
clause in the original --grow command.  You will need that file.

[trim /]

> How should I proceed? I'm far enough out of my depth that I'm hesitant
> to try anything for fear of causing more damage. Should I update my
> mdadm.conf to have num-devices=6 and see if it sorts itself out?

No.

> Try to force assemble the 5 drives with superblocks?

Yes, but see below.

> Create a "new" array out of them?

Absolutely not.

> Any input would be greatly appreciated.

Modern mdadm should be able to force assemble this and continue without
problems.  Rather than operate within a questionable environment, I
would strongly encourage you to perform the forced assembly with a
recent live cd.  I personally use "SystemRescueCD", and I know it has
the appropriate kernel support and tools.

But.  You need to share more information about your hardware problems.
Dmesg, etc.  There are commonly-encountered configuration problems that
appear to be mysterious drive failures.  If you know all about error
recovery control, please elaborate.  Otherwise, please share the output
of "smartctl -x /dev/sdX" for all of your member devices.

Phil

Re: RAID6 growing interrupted, array won't assemble or resume growing

[Nic Wolfe]

My original post had an error - sda is my boot drive, it's not part of
the array. The 6th drive is as follows:

/dev/sdg:
          Magic : a92b4efc
        Version : 00.91.00
           UUID : 4672ced4:81401dbc:52723fc8:3fe02f5a (local to host MidgetNAS)
  Creation Time : Wed Jun  2 21:11:18 2010
     Raid Level : raid6
  Used Dev Size : 1953431488 (1862.94 GiB 2000.31 GB)
     Array Size : 7813725952 (7451.75 GiB 8001.26 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 1

  Reshape pos'n : 677888 (662.11 MiB 694.16 MB)
  Delta Devices : 1 (5->6)

    Update Time : Mon Oct 22 21:08:29 2012
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 4
  Spare Devices : 0
       Checksum : 146beaa7 - correct
         Events : 0.1323362

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     5      65        0        5      active sync

   0     0       0        0        0      removed
   1     1       0        0        1      faulty removed
   2     2       0        0        2      faulty removed
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
   5     5      65        0        5      active sync



> Your reshape is barely started.  Presumably you specified a --backup
> clause in the original --grow command.  You will need that file.

Embarrassingly I have no backup file from the grow operation.

> Modern mdadm should be able to force assemble this and continue without
> problems.  Rather than operate within a questionable environment, I
> would strongly encourage you to perform the forced assembly with a
> recent live cd.  I personally use "SystemRescueCD", and I know it has
> the appropriate kernel support and tools.
>
> But.  You need to share more information about your hardware problems.
> Dmesg, etc.  There are commonly-encountered configuration problems that
> appear to be mysterious drive failures.  If you know all about error
> recovery control, please elaborate.  Otherwise, please share the output
> of "smartctl -x /dev/sdX" for all of your member devices.

My drives are connected to the machine through a poorly supported old
RAID card (rr2522) which required me to build the driver into my
kernel, so I don't think a live cd will work. I don't have enough SATA
slots in the machine to connect them all without it.

If it isn't obvious by now, I definitely don't know anything about
error recovery control.

Since it's a RAID card and not just a hba each drive is presented to
the OS as a single drive JBOD array which means the OS doesn't see any
SMART info. I can see (at least some) SMART information through my
RAID card admin console and it claims they're all fine.

I am not having hardware problems at the moment, I only encountered
them when I had 16 drives running through the RAID card (I have
another 10 drive array). With the other array disconnected the card
seems to be behaving - there's nothing suspicious that I can see in
dmesg.

In the meantime I will see if I can put together a machine with 6 SATA
ports and attempt to hook the drives up directly rather than through
the RAID card so I can use a live CD and get the SMART information for
you.

Thanks for the reply,

Nic

Re: RAID6 growing interrupted, array won't assemble or resume growing

[Phil Turmel]

On 06/07/2013 12:15 AM, Nic Wolfe wrote:
> My original post had an error - sda is my boot drive, it's not part of
> the array. The 6th drive is as follows:
> 
> /dev/sdg:

[trim /]

>       Number   Major   Minor   RaidDevice State
> this     5      65        0        5      active sync

Before you move drives around, please spend some time correlating drive
serial numbers to the "RaidDevice" number for each.  Just in case.  You
may be able to get everything you need from /dev/disk/by-id/, but with a
weird driver, I not sure.

>> Your reshape is barely started.  Presumably you specified a --backup
>> clause in the original --grow command.  You will need that file.
> 
> Embarrassingly I have no backup file from the grow operation.

mdadm should have halted if it knew it was needed.  You may be ok.

>> Modern mdadm should be able to force assemble this and continue without
>> problems.  Rather than operate within a questionable environment, I
>> would strongly encourage you to perform the forced assembly with a
>> recent live cd.  I personally use "SystemRescueCD", and I know it has
>> the appropriate kernel support and tools.
>>
>> But.  You need to share more information about your hardware problems.
>> Dmesg, etc.  There are commonly-encountered configuration problems that
>> appear to be mysterious drive failures.  If you know all about error
>> recovery control, please elaborate.  Otherwise, please share the output
>> of "smartctl -x /dev/sdX" for all of your member devices.
> 
> My drives are connected to the machine through a poorly supported old
> RAID card (rr2522) which required me to build the driver into my
> kernel, so I don't think a live cd will work. I don't have enough SATA
> slots in the machine to connect them all without it.

Hmmm.  That's ugly.

> If it isn't obvious by now, I definitely don't know anything about
> error recovery control.

Well, search the archive for terms like "scterc", "ure", and/or
"timeout".  You'll need to address the issue if you don't have
enterprise drives.

> Since it's a RAID card and not just a hba each drive is presented to
> the OS as a single drive JBOD array which means the OS doesn't see any
> SMART info. I can see (at least some) SMART information through my
> RAID card admin console and it claims they're all fine.

Drives with pending unrecoverable read errors won't make it through a
rebuild, but show "PASSED" to smart.  The details matter.

> I am not having hardware problems at the moment, I only encountered
> them when I had 16 drives running through the RAID card (I have
> another 10 drive array). With the other array disconnected the card
> seems to be behaving - there's nothing suspicious that I can see in
> dmesg.

That sounds like insufficient power.

> In the meantime I will see if I can put together a machine with 6 SATA
> ports and attempt to hook the drives up directly rather than through
> the RAID card so I can use a live CD and get the SMART information for
> you.

Good plan, but do get the serial numbers vs. array role info.  Probably
should finish the reshape in such a machine, too.

Phil

Re: RAID6 growing interrupted, array won't assemble or resume growing

[Nic Wolfe]

> Before you move drives around, please spend some time correlating drive
> serial numbers to the "RaidDevice" number for each.  Just in case.  You
> may be able to get everything you need from /dev/disk/by-id/, but with a
> weird driver, I not sure.
>

Done, thanks.

>
> Well, search the archive for terms like "scterc", "ure", and/or
> "timeout".  You'll need to address the issue if you don't have
> enterprise drives.
>

[trim /]

>
> Drives with pending unrecoverable read errors won't make it through a
> rebuild, but show "PASSED" to smart.  The details matter.
>
>> I am not having hardware problems at the moment, I only encountered
>> them when I had 16 drives running through the RAID card (I have
>> another 10 drive array). With the other array disconnected the card
>> seems to be behaving - there's nothing suspicious that I can see in
>> dmesg.
>
> That sounds like insufficient power.
>
>> In the meantime I will see if I can put together a machine with 6 SATA
>> ports and attempt to hook the drives up directly rather than through
>> the RAID card so I can use a live CD and get the SMART information for
>> you.
>
> Good plan, but do get the serial numbers vs. array role info.  Probably
> should finish the reshape in such a machine, too.

OK I have built a PC with properly supported hardware that I can now
hook all my drives up to directly. Unfortunately because they were
being managed by the RAID card before as single drive JBOD arrays they
do not show up as md devices on the new PC, so I will have to attempt
to get this array running on the old PC with the old RAID/JBOD setup.

On the new PC I can see the SMART info but since the drives will have
to be managed by the RAID card anyway I'm not sure if it is prudent to
mess with the settings. It does sound like a good idea to maybe
increase the timeouts on the OS though ("echo 180
>/sys/block/sd[bcdefg]/device/timeout"). And hopefully my problems
were just power in the first place?

In case it helps though:

sdb: http://pastebin.com/btrpcMF0
sdc: http://pastebin.com/kdcrf9mY
sdd: http://pastebin.com/S0nj9Ma9
sde: http://pastebin.com/G3vUZf92
sdf: http://pastebin.com/0P2LbDQq
sdg: http://pastebin.com/8q4vuU1a

Finally, I don't actually necessarily need to finish the reshape; I
just want to get the data off. Can I assemble the array as degraded
and just copy the data to a new location without finishing the
reshape? It seems that'd save a whole lot of writing and potential for
problems. If I can get the data off I can then blank the old drives
and start over properly without the sucky RAID card and with the
proper smart/erc settings. If that's possible how would I go about
doing that other than just trying to --assemble?

Thanks,

Nic

Re: RAID6 growing interrupted, array won't assemble or resume growing

[Phil Turmel]

On 06/19/2013 02:21 AM, Nic Wolfe wrote:

>> Well, search the archive for terms like "scterc", "ure", and/or
>> "timeout".  You'll need to address the issue if you don't have
>> enterprise drives.
>>
> 
> [trim /]

You trimmed here...  Per the linked reports, you have this problem.

>> Drives with pending unrecoverable read errors won't make it through a
>> rebuild, but show "PASSED" to smart.  The details matter.

Drive /dev/sdc has pending relocations.  Those are bad sectors that need
to be rewritten to either fix them or relocate them.  You are unable to
do that with your raid card because /dev/sdc does not support error
recovery control.

In fact, only your /dev/sdd supports it, and as a desktop drive, it
defaults to "Off" on every power cycle.

[trim /]

> OK I have built a PC with properly supported hardware that I can now
> hook all my drives up to directly. Unfortunately because they were
> being managed by the RAID card before as single drive JBOD arrays they
> do not show up as md devices on the new PC, so I will have to attempt
> to get this array running on the old PC with the old RAID/JBOD setup.

Some raid cards use part of the drive space to write management data,
and show a smaller drive to the OS.  If this is your case, you can
manually set up device mapper to expose the smaller areas as devices,
and build the MD array from those.  Detailed information about your
device sizes with and without the raid card would be needed.

> On the new PC I can see the SMART info but since the drives will have
> to be managed by the RAID card anyway I'm not sure if it is prudent to
> mess with the settings. It does sound like a good idea to maybe
> increase the timeouts on the OS though ("echo 180
>> /sys/block/sd[bcdefg]/device/timeout"). And hopefully my problems
> were just power in the first place?

No.  /dev/sdc is your problem.  There might be more, but they aren't
exposed in SMART yet.

> In case it helps though:
> 
> sdb: http://pastebin.com/btrpcMF0
> sdc: http://pastebin.com/kdcrf9mY
> sdd: http://pastebin.com/S0nj9Ma9
> sde: http://pastebin.com/G3vUZf92
> sdf: http://pastebin.com/0P2LbDQq
> sdg: http://pastebin.com/8q4vuU1a

As noted above, you will not be able to use /dev/sdc with the raid card.
 You could try leaving it out and taking a backup with the degraded
array.  That is what I would recommend first.

> Finally, I don't actually necessarily need to finish the reshape; I
> just want to get the data off. Can I assemble the array as degraded
> and just copy the data to a new location without finishing the
> reshape? It seems that'd save a whole lot of writing and potential for
> problems. If I can get the data off I can then blank the old drives
> and start over properly without the sucky RAID card and with the
> proper smart/erc settings. If that's possible how would I go about
> doing that other than just trying to --assemble?

Second choice is to figure out the "dmsetup" details to run without the
raid card, set the driver timeouts to 180 for all but /dev/sdd.  Turn on
scterc for that one.  Then let MD assemble and continue.  And take your
backup.

Phil

Re: RAID6 growing interrupted, array won't assemble or resume growing

[Nic Wolfe]

On Wed, Jun 19, 2013 at 12:36 PM, Phil Turmel <philip@turmel.org> wrote:

> As noted above, you will not be able to use /dev/sdc with the raid card.
>  You could try leaving it out and taking a backup with the degraded
> array.  That is what I would recommend first.

OK, will do. Simply "mdadm --assemble /dev/md1 /dev/sdb /dev/sdd
/dev/sde /dev/sdf /dev/sdg"?

Thanks so much for all the help so far.

Nic

Re: RAID6 growing interrupted, array won't assemble or resume growing

[Nic Wolfe]

On Wed, Jun 19, 2013 at 12:36 PM, Phil Turmel <philip@turmel.org> wrote:
> On 06/19/2013 02:21 AM, Nic Wolfe wrote:

> As noted above, you will not be able to use /dev/sdc with the raid card.
>  You could try leaving it out and taking a backup with the degraded
> array.  That is what I would recommend first.

midgetspy@MidgetNAS:~$ sudo mdadm --assemble /dev/md1 /dev/sdd
/dev/sdb /dev/sde /dev/sdf /dev/sdg
mdadm: Failed to restore critical section for reshape, sorry.

Some googling indicated that my mdadm version (2.6.3) has a known
issue with reshapes so I got the 3.2.6 source and built it and tried
again:

midgetspy@MidgetNAS:~/mdadm-3.2.6$ sudo ./mdadm --assemble /dev/md1
/dev/sdd /dev/sdb /dev/sde /dev/sdf /dev/sdg
mdadm: /dev/md1 assembled from 1 drive - not enough to start the array.
midgetspy@MidgetNAS:~/mdadm-3.2.6$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md1 : inactive sdg[5](S) sdf[4](S) sde[3](S) sdb[2](S) sdd[0](S)
      9767157440 blocks super 0.91

unused devices: <none>

mdadm --examine: http://pastebin.com/FRQDXDLB

I'm not sure where to go from here.

Thanks,

Nic