Raid5 reshape stuck at 0% - SuSE leap 42.1

Raid5 reshape stuck at 0% - SuSE leap 42.1

[Wols Lists]

Yes I know this is a common occurrence, but it's bitten my test system.

I've now got an array converting from raid1 to raid5, that will build
but won't run, and won't grow.

I suspect it's a similar problem to the one that bit Fedora - in that
"mdadm --grow --continue" comes back with no error messages, but I can
find no evidence that a background mdadm is actually running. As
mentioned earlier, AppArmor is disabled, so I don't know why it won't
work. Any ideas?

I also can't run the array :-(

Note that it's a test vm specifically for playing with raid, so no data
is at stake, but equally, these things shouldn't fail and land you with
a non-working array (and yes, I have downloaded the latest mdadm -
unfortunately I have been mix-n-matching the distro and bleeding-edge
versions).

Cheers,
Wol


/dev/md127:
        Version : 1.2
  Creation Time : Fri Aug  5 18:16:24 2016
     Raid Level : raid5
  Used Dev Size : 8380416 (7.99 GiB 8.58 GB)
   Raid Devices : 3
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun Aug  7 01:20:23 2016
          State : active, Not Started
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

  Delta Devices : 1, (2->3)

           Name : kanga:testarray  (local to host kanga)
           UUID : cf52ebc0:886a35cd:688274b4:3f16096c
         Events : 157

    Number   Major   Minor   RaidDevice State
       4       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       3       8       64        2      active sync   /dev/sde

       2       8       48        -      spare   /dev/sdd



/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x44
     Array UUID : cf52ebc0:886a35cd:688274b4:3f16096c
           Name : kanga:testarray  (local to host kanga)
  Creation Time : Fri Aug  5 18:16:24 2016
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 16760832 (7.99 GiB 8.58 GB)
     Array Size : 16760832 (15.98 GiB 17.16 GB)
    Data Offset : 16384 sectors
     New Offset : 16128 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : e9e6426d:d33b9c14:b8bce7e8:780f848e

  Reshape pos'n : 0
  Delta Devices : 1 (2->3)

    Update Time : Sun Aug  7 01:20:23 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 41eb0fd7 - correct
         Events : 157

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x44
     Array UUID : cf52ebc0:886a35cd:688274b4:3f16096c
           Name : kanga:testarray  (local to host kanga)
  Creation Time : Fri Aug  5 18:16:24 2016
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 16760832 (7.99 GiB 8.58 GB)
     Array Size : 16760832 (15.98 GiB 17.16 GB)
    Data Offset : 16384 sectors
     New Offset : 16128 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 9566866f:cc4cca0d:d8b54faf:f44e2e97

  Reshape pos'n : 0
  Delta Devices : 1 (2->3)

    Update Time : Sun Aug  7 01:20:23 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : c6ed915 - correct
         Events : 157

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x44
     Array UUID : cf52ebc0:886a35cd:688274b4:3f16096c
           Name : kanga:testarray  (local to host kanga)
  Creation Time : Fri Aug  5 18:16:24 2016
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 16760832 (7.99 GiB 8.58 GB)
     Array Size : 16760832 (15.98 GiB 17.16 GB)
    Data Offset : 16384 sectors
     New Offset : 16128 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : fed79c82:8510d13a:496617f5:c1381778

  Reshape pos'n : 0
  Delta Devices : 1 (2->3)

    Update Time : Sun Aug  7 01:20:23 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 733ca876 - correct
         Events : 157

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : spare
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : cf52ebc0:886a35cd:688274b4:3f16096c
           Name : kanga:testarray  (local to host kanga)
  Creation Time : Fri Aug  5 18:16:24 2016
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 16760832 (7.99 GiB 8.58 GB)
     Array Size : 16760832 (15.98 GiB 17.16 GB)
    Data Offset : 16384 sectors
   Super Offset : 8 sectors
   Unused Space : before=16296 sectors, after=0 sectors
          State : clean
    Device UUID : 51271324:1b29e46a:521658dd:5747e61e

  Reshape pos'n : 0
  Delta Devices : 1 (2->3)

    Update Time : Sun Aug  7 01:20:23 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : d3d5cfbd - correct
         Events : 157

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 2
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)

Re: Raid5 reshape stuck at 0% - SuSE leap 42.1

[Mikael Abrahamsson]

On Sun, 7 Aug 2016, Wols Lists wrote:

> I also can't run the array :-(

What does "cat /proc/mdstat" say?

Have you tried "--continue"?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Raid5 reshape stuck at 0% - SuSE leap 42.1

[Wols Lists]

On 07/08/16 10:54, Mikael Abrahamsson wrote:
> On Sun, 7 Aug 2016, Wols Lists wrote:
> 
>> I also can't run the array :-(
> 
> What does "cat /proc/mdstat" say?

The vm is shut down right now, but it said inactive, four drives,
[UUU_], iirc. I'll post it when I get it back up. Didn't mention raid
type, so that could be messed up somewhere.

Two drives are mirrored, it should be rebuilding on to third, and a
fourth is spare (ready to rebuild into raid6)
> 
> Have you tried "--continue"?
> 
The last paragraph before the one you quoted said "yes". Actually,
multiple times.

Cheers,
Wol

Re: Raid5 reshape stuck at 0% - SuSE leap 42.1

[Wols Lists]

On 07/08/16 12:21, Wols Lists wrote:
> On 07/08/16 10:54, Mikael Abrahamsson wrote:
>> > On Sun, 7 Aug 2016, Wols Lists wrote:
>> > 
>>> >> I also can't run the array :-(
>> > 
>> > What does "cat /proc/mdstat" say?
> The vm is shut down right now, but it said inactive, four drives,
> [UUU_], iirc. I'll post it when I get it back up. Didn't mention raid
> type, so that could be messed up somewhere.
> 
> Two drives are mirrored, it should be rebuilding on to third, and a
> fourth is spare (ready to rebuild into raid6)

/dev/sdd seems to have disappeared - maybe I removed it ... I've been
mucking about a fair bit ...

Note that "mdadm" is the version that came with the system, "./mdadm" is
the latest version downloaded via git.

And being SuSE, with systemd, "--continue" is supposed to fire off a
background mdadm, I believe. ps tells me that isn't happening...

kanga:/home/anthony # cd mdadm/
kanga:/home/anthony/mdadm # ./mdadm -D /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Fri Aug  5 18:16:24 2016
     Raid Level : raid5
  Used Dev Size : 8380416 (7.99 GiB 8.58 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Sun Aug  7 01:20:23 2016
          State : active, Not Started
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

  Delta Devices : 1, (2->3)

           Name : kanga:testarray  (local to host kanga)
           UUID : cf52ebc0:886a35cd:688274b4:3f16096c
         Events : 157

    Number   Major   Minor   RaidDevice State
       4       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       3       8       64        2      active sync   /dev/sde
kanga:/home/anthony/mdadm # mdadm /dev/md127 --grow --continue
kanga:/home/anthony/mdadm # ps -fea | grep mdadm
root      1953  1855  0 13:02 pts/0    00:00:00 grep --color=auto mdadm
kanga:/home/anthony/mdadm # cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : inactive sdc[1] sde[3] sdb[4]
      25141248 blocks super 1.2

unused devices: <none>
kanga:/home/anthony/mdadm # ./mdadm /dev/md127 --grow --continue
kanga:/home/anthony/mdadm # ps -fea | grep mdadm
root      1967  1855  0 13:03 pts/0    00:00:00 grep --color=auto mdadm
kanga:/home/anthony/mdadm #

Re: Raid5 reshape stuck at 0% - SuSE leap 42.1

[Mikael Abrahamsson]

On Sun, 7 Aug 2016, Wols Lists wrote:

Please --stop it, then --assemble with verbose output and include output 
from that command plus anything in dmesg at the same time. What happens 
when you then try to --run it?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Raid5 reshape stuck at 0% - SuSE leap 42.1

[Wols Lists]

On 07/08/16 15:31, Mikael Abrahamsson wrote:
> On Sun, 7 Aug 2016, Wols Lists wrote:
> 
> Please --stop it, then --assemble with verbose output and include output
> from that command plus anything in dmesg at the same time. What happens
> when you then try to --run it?
> 
Output below ...


kanga:/home/anthony/mdadm # ./mdadm --stop /dev/md127
mdadm: stopped /dev/md127
kanga:/home/anthony/mdadm # mdadm -v --assemble /dev/md127 /dev/sdb
/dev/sdc /dev/sdd /dev/sde
mdadm: looking for devices for /dev/md127
mdadm: /dev/sdb is identified as a member of /dev/md127, slot 0.
mdadm: /dev/sdc is identified as a member of /dev/md127, slot 1.
mdadm: /dev/sdd is identified as a member of /dev/md127, slot -1.
mdadm: /dev/sde is identified as a member of /dev/md127, slot 2.
mdadm: added /dev/sdc to /dev/md127 as 1
mdadm: added /dev/sde to /dev/md127 as 2
mdadm: added /dev/sdd to /dev/md127 as -1
mdadm: added /dev/sdb to /dev/md127 as 0
mdadm: failed to RUN_ARRAY /dev/md127: Invalid argument
kanga:/home/anthony/mdadm #



[17372.305472] md: md127 stopped.
[17372.305483] md: unbind<sdc>
[17372.312124] md: export_rdev(sdc)
[17372.312198] md: unbind<sde>
[17372.317792] md: export_rdev(sde)
[17372.317868] md: unbind<sdb>
[17372.320344] md: export_rdev(sdb)
[17431.701768] md: md127 stopped.
[17431.710474] md: bind<sdc>
[17431.711203] md: bind<sde>
[17431.712843] md: bind<sdd>
[17431.713735] md: bind<sdb>
[17431.752149] md/raid:md127: reshape_position too early for
auto-recovery - aborting.
[17431.752155] md: pers->run() failed ...
[17431.752289] md: md127 stopped.
[17431.752297] md: unbind<sdb>
[17431.760544] md: export_rdev(sdb)
[17431.760602] md: unbind<sdd>
[17431.772124] md: export_rdev(sdd)
[17431.772144] md: unbind<sde>
[17431.780125] md: export_rdev(sde)
[17431.780144] md: unbind<sdc>
[17431.792163] md: export_rdev(sdc)
[17431.909807] md: bind<sde>
[17431.942187] md: bind<sdc>
[17431.962809] md: bind<sdd>
[17431.989609] md: bind<sdb>
[17432.026378] md/raid:md127: reshape_position too early for
auto-recovery - aborting.
[17432.026385] md: pers->run() failed ...
anthony@kanga:/mnt/anthony>

Re: Raid5 reshape stuck at 0% - SuSE leap 42.1

[Mikael Abrahamsson]

On Sun, 7 Aug 2016, Wols Lists wrote:

> [17431.752149] md/raid:md127: reshape_position too early for
> auto-recovery - aborting.

Does the hint in the below URL help?

http://www.spinics.net/lists/raid/msg52319.html

(to use "--update=revert-reshape")

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Raid5 reshape stuck at 0% - SuSE leap 42.1

[Wols Lists]

On 07/08/16 18:24, Mikael Abrahamsson wrote:
> On Sun, 7 Aug 2016, Wols Lists wrote:
> 
>> [17431.752149] md/raid:md127: reshape_position too early for
>> auto-recovery - aborting.
> 
> Does the hint in the below URL help?
> 
> http://www.spinics.net/lists/raid/msg52319.html
> 
> (to use "--update=revert-reshape")
> 
I have tried it - I'm just looking at the url now to have another go ...

Cheers,
Wol

Re: Raid5 reshape stuck at 0% - SuSE leap 42.1

[Wols Lists]

On 07/08/16 18:24, Mikael Abrahamsson wrote:
> On Sun, 7 Aug 2016, Wols Lists wrote:
> 
>> [17431.752149] md/raid:md127: reshape_position too early for
>> auto-recovery - aborting.
> 
> Does the hint in the below URL help?
> 
> http://www.spinics.net/lists/raid/msg52319.html
> 
> (to use "--update=revert-reshape")
> 
Okay - it's worked - as in I've got a working array back - a two-drive
raid 5 array with two spares.

So hwo do I now get it to rebuild successfully to give me a working full
raid 5?

I'm guessing it should be "mdadm --grow --raid-devices=3
--backup-file=whatever", but I think I've already tried that, and that's
what got me into this mess :-)

I'm trying to get access to the linux-raid wiki, so once I've got this
working, and I've got the access, all this stuff is going to be written
up and posted to the wiki :-)

Cheers,
Wol

Re: Raid5 reshape stuck at 0% - SuSE leap 42.1

[Mikael Abrahamsson]

On Sun, 7 Aug 2016, Wols Lists wrote:

> I'm guessing it should be "mdadm --grow --raid-devices=3
> --backup-file=whatever", but I think I've already tried that, and that's
> what got me into this mess :-)

Try without backup file (it shouldn't be needed when growing under these 
circumstances). If it doesn't start rebuilding after issuing --grow, try 
--continue when the array is actually up and running (don't stop it when 
it's "frozen" and says rebuilding but doesn't actually show any rebuilding 
going on). That has worked for others.

Use latest git version of mdadm.

Please post output from all commands and dmesg if it still fails.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Raid5 reshape stuck at 0% - SuSE leap 42.1

[Wols Lists]

On 07/08/16 19:05, Mikael Abrahamsson wrote:
> On Sun, 7 Aug 2016, Wols Lists wrote:
> 
>> I'm guessing it should be "mdadm --grow --raid-devices=3
>> --backup-file=whatever", but I think I've already tried that, and that's
>> what got me into this mess :-)
> 
> Try without backup file (it shouldn't be needed when growing under these
> circumstances). If it doesn't start rebuilding after issuing --grow, try
> --continue when the array is actually up and running (don't stop it when
> it's "frozen" and says rebuilding but doesn't actually show any
> rebuilding going on). That has worked for others.
> 
> Use latest git version of mdadm.
> 
> Please post output from all commands and dmesg if it still fails.
> 
Guess what ...

Note that I think this dmesg stuff overlaps with the last lot, namely
the start of this is the tail end of the array starting successfully
last time.

I'm using ./mdadm, so that's the version I downloaded from Neil's
repository yesterday ... I'll investigate the source when I get the
chance (wives!!! :-)

So is this virtualbox and fuse playing up? Dunno ... (this vm is
mono-processor - shouldn't make any difference ...)

Cheers,
Wol



mdadm: /dev/md127 has been started with 2 drives and 2 spares.
kanga:/home/anthony/mdadm # mount /dev/md127 /mnt/testarray/
kanga:/home/anthony/mdadm # ./madm /dev/md127 --grow --raid-devices=3
bash: ./madm: No such file or directory
kanga:/home/anthony/mdadm # ./mdadm /dev/md127 --grow --raid-devices=3
mdadm: Failed to initiate reshape!
kanga:/home/anthony/mdadm # ./mdadm /dev/md127 --grow --continue
kanga:/home/anthony/mdadm #


>
[21390.529578] md/raid:md127: device sdb operational as raid disk 0
[21390.529618] md/raid:md127: device sdc operational as raid disk 1
[21390.531258] md/raid:md127: allocated 2250kB
[21390.534250] md/raid:md127: raid level 5 active with 2 out of 2
devices, algorithm 2
[21390.534259] RAID conf printout:
[21390.534263]  --- level:5 rd:2 wd:2
[21390.534269]  disk 0, o:1, dev:sdb
[21390.534273]  disk 1, o:1, dev:sdc
[21390.534419] md127: detected capacity change from 0 to 8581545984
[21390.539650] RAID conf printout:
[21390.539682]  --- level:5 rd:2 wd:2
[21390.539690]  disk 0, o:1, dev:sdb
[21390.539694]  disk 1, o:1, dev:sdc
[21390.539698] RAID conf printout:
[21390.539701]  --- level:5 rd:2 wd:2
[21390.539704]  disk 0, o:1, dev:sdb
[21390.539707]  disk 1, o:1, dev:sdc
[21435.972237] EXT4-fs (md127): mounted filesystem with ordered data
mode. Opts: (null)
[22157.683863] device-mapper: uevent: version 1.0.3
[22157.686018] device-mapper: ioctl: 4.31.0-ioctl (2015-3-12)
initialised: dm-devel@redhat.com
[22165.688223] fuse init (API version 7.23)
[22243.094448] 06:10:19.795789 control  Error: Getting host message
failed with VERR_INTERRUPTED
[22244.094733] 06:10:20.796092 control  Guest control service stopped
[22244.094789] 06:10:20.796207 control  Guest control worker returned
with rc=VERR_INTERRUPTED
[22244.095871] 06:10:20.797229 main     Session 0 is about to close ...
[22244.095922] 06:10:20.797343 main     Stopping all guest processes ...
[22244.095956] 06:10:20.797379 main     Closing all guest files ...
[22244.099090] 06:10:20.800461 main     Ended.
[22244.641855] VBoxService 5.0.24_SUSE r108355 (verbosity: 0)
linux.amd64 (openSUSE Buildservice) release log
00:00:00.000257 main     Log opened 2016-08-07T18:07:43.777555000Z
[22244.641971] 00:00:00.000481 main     OS Product: Linux
[22244.642016] 00:00:00.000526 main     OS Release: 4.1.15-8-default
[22244.642056] 00:00:00.000564 main     OS Version: #1 SMP PREEMPT Wed
Jan 20 16:41:00 UTC 2016 (0e3b3ab)
[22244.642206] 00:00:00.000604 main     Executable: /usr/sbin/VBoxService
00:00:00.000605 main     Process ID: 12443
00:00:00.000606 main     Package type: LINUX_64BITS_GENERIC (OSE)
[22244.643245] 00:00:00.001736 main     5.0.24_SUSE r108355 started.
Verbose level = 0
[22567.452608] SFW2-INext-DROP-DEFLT IN=eth0 OUT=
MAC=08:00:27:0b:0f:57:52:54:00:12:35:02:08:00 SRC=10.0.2.2 DST=10.0.2.15
LEN=576 TOS=0x10 PREC=0x00 TTL=64 ID=2252 PROTO=UDP SPT=67 DPT=68 LEN=556
[25196.727211] RAID conf printout:
[25196.727217]  --- level:5 rd:2 wd:2
[25196.727238]  disk 0, o:1, dev:sdb
[25196.727240]  disk 1, o:1, dev:sdc
[25196.727246] RAID conf printout:
[25196.727248]  --- level:5 rd:2 wd:2
[25196.727250]  disk 0, o:1, dev:sdb
[25196.727251]  disk 1, o:1, dev:sdc
anthony@kanga:/mnt/anthony>

Re: Raid5 reshape stuck at 0% - SuSE leap 42.1

[Mikael Abrahamsson]

On Sun, 7 Aug 2016, Wols Lists wrote:

> Note that I think this dmesg stuff overlaps with the last lot, namely 
> the start of this is the tail end of the array starting successfully 
> last time.

It helps if you supply output of /proc/mdstat before each operation, and 
also adding verbose output to mdadm command.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Raid5 reshape stuck at 0% - SuSE leap 42.1

[Wols Lists]

On 08/08/16 12:40, Mikael Abrahamsson wrote:
> On Sun, 7 Aug 2016, Wols Lists wrote:
> 
>> Note that I think this dmesg stuff overlaps with the last lot, namely
>> the start of this is the tail end of the array starting successfully
>> last time.
> 
> It helps if you supply output of /proc/mdstat before each operation, and
> also adding verbose output to mdadm command.
> 
Okay. Just tried to do this - and do Neil's thing where I was trying to
reduce the number of raid devices ... I now have a wedged 2-device raid
5 that I can't revert back to raid 1, or set off the required reshape.
Mikael - your --update=revert-reshape that worked fine last time, now
refuses to work ... :-( so I can't try Neil's --raid-devices=2 because I
can't get a clean full-working-order array. (The array is working fine,
so if it was a real live array I wouldn't be worried about losing
anything, but a wedged array is a wedged array - not good!)

As before, OS = SuSE Leap 42.1, "mdadm" is what comes with the OS,
"./mdadm" is Neil's git tree (a week or so old).

Attached is my xterm trace and the associated output from dmesg.

kanga:/home/anthony/mdadm # mdadm --stop /dev/md127
mdadm: stopped /dev/md127
kanga:/home/anthony/mdadm # ./mdadm --assemble /dev/md127 --verbose
--force --update=revert-reshape --invalid-backup
--backup-file=../raidbackup /dev/sdb /dev/sdc /dev/sdd /dev/sde
mdadm: looking for devices for /dev/md127
mdadm: No active reshape to revert on /dev/sdb
kanga:/home/anthony/mdadm # ./mdadm --assemble /dev/md127 --verbose
--force --update=revert-reshape --invalid-backup
--backup-file=../raidbackup /dev/sdc /dev/sdd /dev/sde     mdadm:
looking for devices for /dev/md127
mdadm: Merging with already-assembled /dev/md/testarray
mdadm: No active reshape to revert on /dev/sdb
kanga:/home/anthony/mdadm # mdadm --assemble --scan
mdadm: Merging with already-assembled /dev/md/testarray
mdadm: /dev/md/testarray has been started with 2 drives and 2 spares.
kanga:/home/anthony/mdadm # mdadm - D /dev/md127
mdadm: An option must be given to set the mode before a second device
       (D) is listed
kanga:/home/anthony/mdadm # mdadm -D /dev/md127
/dev/md127:
        Version : 1.2
  Creation Time : Fri Aug  5 18:16:24 2016
     Raid Level : raid5
     Array Size : 8380416 (7.99 GiB 8.58 GB)
  Used Dev Size : 8380416 (7.99 GiB 8.58 GB)
   Raid Devices : 2
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Tue Aug 16 11:32:52 2016
          State : clean
 Active Devices : 2
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 2

         Layout : left-symmetric
     Chunk Size : 64K

           Name : kanga:testarray  (local to host kanga)
           UUID : cf52ebc0:886a35cd:688274b4:3f16096c
         Events : 160

    Number   Major   Minor   RaidDevice State
       4       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc

       2       8       48        -      spare   /dev/sdd
       3       8       64        -      spare   /dev/sde
kanga:/home/anthony/mdadm # mdadm --grow /dev/md127 --continue
kanga:/home/anthony/mdadm # cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd[2](S) sde[3](S) sdc[1] sdb[4]
      8380416 blocks super 1.2 level 5, 64k chunk, algorithm 2 [2/2] [UU]

unused devices: <none>
kanga:/home/anthony/mdadm # mdadm --stop /dev/md127
mdadm: stopped /dev/md127
kanga:/home/anthony/mdadm # ./mdadm --assemble /dev/md127 --verbose
--force --update=revert-reshape --invalid-backup
--backup-file=../raidbackup /dev/sdb /dev/sdc /dev/sdd /dev/sde
mdadm: looking for devices for /dev/md127
mdadm: No active reshape to revert on /dev/sdb
kanga:/home/anthony/mdadm # ./mdadm --grow /dev/md127 --continue
mdadm: /dev/md127 is not an active md array - aborting
kanga:/home/anthony/mdadm # ./mdadm --assemble --scan
mdadm: Merging with already-assembled /dev/md/testarray
mdadm: /dev/md/testarray has been started with 2 drives and 2 spares.
kanga:/home/anthony/mdadm # ./mdadm --grow /dev/md127 --continue
kanga:/home/anthony/mdadm # cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md127 : active raid5 sdd[2](S) sde[3](S) sdc[1] sdb[4]
      8380416 blocks super 1.2 level 5, 64k chunk, algorithm 2 [2/2] [UU]

unused devices: <none>
kanga:/home/anthony/mdadm # ps -fea | grep mdadm
root      1484  1855  0 11:42 pts/0    00:00:00 grep --color=auto mdadm
root      3586     1  0 09:53 ?        00:00:00 /sbin/mdadm --monitor -d
60 -m root@localhost --scan -c /etc/mdadm.conf
kanga:/home/anthony/mdadm #




00:00:00.000257 main     Log opened 2016-08-07T18:07:43.777555000Z
[22244.641971] 00:00:00.000481 main     OS Product: Linux
[22244.642016] 00:00:00.000526 main     OS Release: 4.1.15-8-default
[22244.642056] 00:00:00.000564 main     OS Version: #1 SMP PREEMPT Wed
Jan 20 16:41:00 UTC 2016 (0e3b3ab)
[22244.642206] 00:00:00.000604 main     Executable: /usr/sbin/VBoxService
00:00:00.000605 main     Process ID: 12443
00:00:00.000606 main     Package type: LINUX_64BITS_GENERIC (OSE)
[22244.643245] 00:00:00.001736 main     5.0.24_SUSE r108355 started.
Verbose level = 0
[22567.452608] SFW2-INext-DROP-DEFLT IN=eth0 OUT=
MAC=08:00:27:0b:0f:57:52:54:00:12:35:02:08:00 SRC=10.0.2.2 DST=10.0.2.15
LEN=576 TOS=0x10 PREC=0x00 TTL=64 ID=2252 PROTO=UDP SPT=67 DPT=68 LEN=556
[25196.727211] RAID conf printout:
[25196.727217]  --- level:5 rd:2 wd:2
[25196.727238]  disk 0, o:1, dev:sdb
[25196.727240]  disk 1, o:1, dev:sdc
[25196.727246] RAID conf printout:
[25196.727248]  --- level:5 rd:2 wd:2
[25196.727250]  disk 0, o:1, dev:sdb
[25196.727251]  disk 1, o:1, dev:sdc
[27195.944364] usb 2-1: USB disconnect, device number 2
[27196.148347] e1000: eth0 NIC Link is Down
[27196.572099] usb 2-1: new full-speed USB device number 3 using ohci-pci
[27196.832714] usb 2-1: New USB device found, idVendor=80ee, idProduct=0021
[27196.832721] usb 2-1: New USB device strings: Mfr=1, Product=3,
SerialNumber=0
[27196.832723] usb 2-1: Product: USB Tablet
[27196.832725] usb 2-1: Manufacturer: VirtualBox
[27196.841815] input: VirtualBox USB Tablet as
/devices/pci0000:00/0000:00:06.0/usb2/2-1/2-1:1.0/0003:80EE:0021.0002/input/input9
[27196.842185] hid-generic 0003:80EE:0021.0002: input,hidraw0: USB HID
v1.10 Mouse [VirtualBox USB Tablet] on usb-0000:00:06.0-1/input0
[27202.148448] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: RX
[27202.152800] SFW2-INext-DROP-DEFLT IN=eth0 OUT=
MAC=08:00:27:0b:0f:57:52:54:00:12:35:02:08:00 SRC=10.0.2.2 DST=10.0.2.15
LEN=576 TOS=0x10 PREC=0x00 TTL=64 ID=0 PROTO=UDP SPT=67 DPT=68 LEN=556
[27265.599076] BTRFS info (device sda2): relocating block group
16202596352 flags 36
[27266.319748] BTRFS info (device sda2): relocating block group
12410945536 flags 34
[27266.439613] BTRFS info (device sda2): relocating block group
16202596352 flags 34
[27266.542733] BTRFS info (device sda2): relocating block group
16236150784 flags 34
[27266.649202] BTRFS info (device sda2): relocating block group
16269705216 flags 34
[27266.757287] BTRFS info (device sda2): relocating block group
12444499968 flags 36
[27271.176254] BTRFS info (device sda2): found 1011 extents
[27271.426455] BTRFS info (device sda2): relocating block group
16303259648 flags 34
[27271.707481] BTRFS info (device sda2): relocating block group
14994636800 flags 36
[27277.442719] BTRFS info (device sda2): found 2417 extents
[27277.862520] BTRFS info (device sda2): relocating block group
13652459520 flags 36
[27283.603832] BTRFS info (device sda2): found 1696 extents
[27366.620131] md127: detected capacity change from 8581545984 to 0
[27366.620131] md: md127 stopped.
[27366.620131] md: unbind<sdb>
[27366.624318] md: export_rdev(sdb)
[27366.624318] md: unbind<sdd>
[27366.632107] md: export_rdev(sdd)
[27366.632137] md: unbind<sde>
[27366.640248] md: export_rdev(sde)
[27366.640248] md: unbind<sdc>
[27366.644324] md: export_rdev(sdc)
[27373.643627] md: md127 stopped.
[27373.775068] md: bind<sdb>
[27437.466705] md: array md127 already has disks!
[27437.467188] md: bind<sdc>
[27437.467593] md: bind<sde>
[27437.467976] md: bind<sdd>
[27437.509156] md/raid:md127: device sdc operational as raid disk 1
[27437.509161] md/raid:md127: device sdb operational as raid disk 0
[27437.509679] md/raid:md127: allocated 2250kB
[27437.520845] md/raid:md127: raid level 5 active with 2 out of 2
devices, algorithm 2
[27437.520850] RAID conf printout:
[27437.520852]  --- level:5 rd:2 wd:2
[27437.520854]  disk 0, o:1, dev:sdb
[27437.520856]  disk 1, o:1, dev:sdc
[27437.521657] md127: detected capacity change from 0 to 8581545984
[27437.523369] RAID conf printout:
[27437.523378]  --- level:5 rd:2 wd:2
[27437.523380]  disk 0, o:1, dev:sdb
[27437.523382]  disk 1, o:1, dev:sdc
[27437.523383] RAID conf printout:
[27437.523384]  --- level:5 rd:2 wd:2
[27437.523384]  disk 0, o:1, dev:sdb
[27437.523385]  disk 1, o:1, dev:sdc
[27587.102558] md127: detected capacity change from 8581545984 to 0
[27587.102569] md: md127 stopped.
[27587.102575] md: unbind<sdd>
[27587.108084] md: export_rdev(sdd)
[27587.108132] md: unbind<sde>
[27587.120239] md: export_rdev(sde)
[27587.120239] md: unbind<sdc>
[27587.128050] md: export_rdev(sdc)
[27587.128069] md: unbind<sdb>
[27587.132042] md: export_rdev(sdb)
[27603.862527] md: md127 stopped.
[27603.963036] md: bind<sdb>
anthony@kanga:/mnt/anthony>