RAID6 grow failed

RAID6 grow failed

[Bryan Bush]

I hope this is the right place to ask this question.  I have at 8
drive RAID 6 array that I wanted to grow to 13 drives (adding 5 more).
 I issued the mdadm command and checked /proc/mdstat and all looked
well.  However at some point in time a disk failed and that hung my
system.  Upon reboot the array is inactive and I can't get it to
reassemble.

/proc/mdstat shows this

md1 : inactive sdp1[11](S) sdi1[3](S) sdd1[7](S) sdr1[13](S)
sdg1[1](S) sdc1[6](S) sdq1[12](S) sdn1[9](S) sdo1[10](S) sdh1[2](S)
sda1[4](S) sdf1[0](S) sdb1[8](S)
      25395674609 blocks super 1.2


If I look at mdadm -E /dev/sdX1 I see most are State active, while
some are State clean.


root@diamond:~# mdadm -E /dev/sd[abcdfghinopqr]1
mdadm: metadata format 01.02 unknown, ignored.
mdadm: metadata format 00.90 unknown, ignored.
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : b7c75aff:703e77dd:8be71623:24cfd5c4

  Reshape pos'n : 1202473536 (1146.77 GiB 1231.33 GB)
  Delta Devices : 5 (8->13)

    Update Time : Tue Mar 27 20:42:37 2012
       Checksum : b892c40c - correct
         Events : 2771

     Chunk Size : 64K

    Array Slot : 4 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_uuUuuuuuuuu 2 failed
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 73f50290:c36a95a1:99de8ad3:2a4e84c8

  Reshape pos'n : 1202473536 (1146.77 GiB 1231.33 GB)
  Delta Devices : 5 (8->13)

    Update Time : Tue Mar 27 20:42:37 2012
       Checksum : c17b9433 - correct
         Events : 2771

     Chunk Size : 64K

    Array Slot : 8 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_uuuUuuuuuuu 2 failed
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : de927013:298f7906:5abdae1e:ccab9188

  Reshape pos'n : 1202473536 (1146.77 GiB 1231.33 GB)
  Delta Devices : 5 (8->13)

    Update Time : Tue Mar 27 20:42:37 2012
       Checksum : b4fe9263 - correct
         Events : 2771

     Chunk Size : 64K

    Array Slot : 6 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_uuuuUuuuuuu 2 failed
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : a258e2c3:e4abf94c:33be105e:fff62610

  Reshape pos'n : 1202473536 (1146.77 GiB 1231.33 GB)
  Delta Devices : 5 (8->13)

    Update Time : Tue Mar 27 20:42:37 2012
       Checksum : 72e7c0f0 - correct
         Events : 2771

     Chunk Size : 64K

    Array Slot : 7 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_uuuuuUuuuuu 2 failed
/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 14c53549:bd43ec01:794d70ff:3ea30a9d

  Reshape pos'n : 1202473536 (1146.77 GiB 1231.33 GB)
  Delta Devices : 5 (8->13)

    Update Time : Tue Mar 27 20:42:37 2012
       Checksum : db7100b9 - correct
         Events : 2771

     Chunk Size : 64K

    Array Slot : 0 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : U_uuuuuuuuuuu 2 failed
/dev/sdg1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 785b908f:e1392eb6:131a836d:4d554339

  Reshape pos'n : 1202143360 (1146.45 GiB 1230.99 GB)
  Delta Devices : 5 (8->13)

    Update Time : Tue Mar 27 20:41:43 2012
       Checksum : e051f831 - correct
         Events : 2768

     Chunk Size : 64K

    Array Slot : 1 (0, 1, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : uUuuuuuuuuuuu 1 failed
/dev/sdh1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : dd08d581:c542403f:33ab15bb:e5a7f122

  Reshape pos'n : 1202473536 (1146.77 GiB 1231.33 GB)
  Delta Devices : 5 (8->13)

    Update Time : Tue Mar 27 20:42:37 2012
       Checksum : 92f0a5ed - correct
         Events : 2771

     Chunk Size : 64K

    Array Slot : 2 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_Uuuuuuuuuuu 2 failed
/dev/sdi1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 6efe22ba:87cc1ffb:a070dbd0:b9020b5e

  Reshape pos'n : 1202473536 (1146.77 GiB 1231.33 GB)
  Delta Devices : 5 (8->13)

    Update Time : Tue Mar 27 20:42:37 2012
       Checksum : d7fd4583 - correct
         Events : 2771

     Chunk Size : 64K

    Array Slot : 3 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_uUuuuuuuuuu 2 failed
/dev/sdn1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : bf54f25f:3a82eb7e:4b852f91:6857c476

  Reshape pos'n : 1202473536 (1146.77 GiB 1231.33 GB)
  Delta Devices : 5 (8->13)

    Update Time : Tue Mar 27 20:42:37 2012
       Checksum : daa5bae6 - correct
         Events : 2771

     Chunk Size : 64K

    Array Slot : 9 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_uuuuuuuuuuU 2 failed
/dev/sdo1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : e239f08a:e56a2abd:18bd6e8d:6e614aa2

  Reshape pos'n : 1202473536 (1146.77 GiB 1231.33 GB)
  Delta Devices : 5 (8->13)

    Update Time : Tue Mar 27 20:42:37 2012
       Checksum : 6ba7ca89 - correct
         Events : 2771

     Chunk Size : 64K

    Array Slot : 10 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_uuuuuuuuuUu 2 failed
/dev/sdp1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : f04f830d:ffc3352f:443c92c0:b03ec412

  Reshape pos'n : 1202473536 (1146.77 GiB 1231.33 GB)
  Delta Devices : 5 (8->13)

    Update Time : Tue Mar 27 20:42:37 2012
       Checksum : 3e3961f - correct
         Events : 2771

     Chunk Size : 64K

    Array Slot : 11 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_uuuuuuuuUuu 2 failed
/dev/sdq1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 5f84b70f:1b8b607b:e2f2cfee:f1fe34f4

  Reshape pos'n : 1202473536 (1146.77 GiB 1231.33 GB)
  Delta Devices : 5 (8->13)

    Update Time : Tue Mar 27 20:42:37 2012
       Checksum : 61f1088b - correct
         Events : 2771

     Chunk Size : 64K

    Array Slot : 12 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_uuuuuuuUuuu 2 failed
/dev/sdr1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 74bb612c:34c2ddf6:4828ed5c:22ea7b38

  Reshape pos'n : 1202473536 (1146.77 GiB 1231.33 GB)
  Delta Devices : 5 (8->13)

    Update Time : Tue Mar 27 20:42:37 2012
       Checksum : ac7c9750 - correct
         Events : 2771

     Chunk Size : 64K

    Array Slot : 13 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_uuuuuuUuuuu 2 failed
root@diamond:~#


Is there anything I can do to get the array back up?
Thanks
-Bryan

Re: RAID6 grow failed

[Mathias Burén]

On 28 March 2012 03:44, Bryan Bush <bbushvt@gmail.com> wrote:
> I hope this is the right place to ask this question.  I have at 8
> drive RAID 6 array that I wanted to grow to 13 drives (adding 5 more).
>  I issued the mdadm command and checked /proc/mdstat and all looked
> well.  However at some point in time a disk failed and that hung my
> system.  Upon reboot the array is inactive and I can't get it to
> reassemble.

<snip>
>
> Is there anything I can do to get the array back up?
> Thanks
> -Bryan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Sorry I can't help you, but that many large HDDs in one array? Can't
help but think that's a bit over-the-top risky! Good luck with the
recovery.

Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID6 grow failed

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1489 bytes --]

On Tue, 27 Mar 2012 22:44:18 -0400 Bryan Bush <bbushvt@gmail.com> wrote:

> I hope this is the right place to ask this question.  I have at 8
> drive RAID 6 array that I wanted to grow to 13 drives (adding 5 more).
>  I issued the mdadm command and checked /proc/mdstat and all looked
> well.  However at some point in time a disk failed and that hung my
> system.  Upon reboot the array is inactive and I can't get it to
> reassemble.
> 
> /proc/mdstat shows this
> 
> md1 : inactive sdp1[11](S) sdi1[3](S) sdd1[7](S) sdr1[13](S)
> sdg1[1](S) sdc1[6](S) sdq1[12](S) sdn1[9](S) sdo1[10](S) sdh1[2](S)
> sda1[4](S) sdf1[0](S) sdb1[8](S)
>       25395674609 blocks super 1.2
> 
> 
> If I look at mdadm -E /dev/sdX1 I see most are State active, while
> some are State clean.
> 
> 
> root@diamond:~# mdadm -E /dev/sd[abcdfghinopqr]1
> mdadm: metadata format 01.02 unknown, ignored.
> mdadm: metadata format 00.90 unknown, ignored.

Hmmm... what do you have in /etc/mdadm.conf??


> 
> Is there anything I can do to get the array back up?

stg1 is the device that failed. so

 mdadm -S /dev/md1
 mdadm -A -f /dev/md1 /dev/sd[abcdefhinopqr]1

should start the array.

Though if the names have changed at all it would be safer to do

  mdadm -Asf /dev/md1 -u fa32e2c5:e7bda20b:32af7c90:c7ee61eb

then mdadm will find the right devices and use them.

When reshape finishes you will need to add sdg1 or a replacement and let it
recover.

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID6 grow failed

[Bryan Bush]

On Wed, Mar 28, 2012 at 12:18 AM, NeilBrown <neilb@suse.de> wrote:
> On Tue, 27 Mar 2012 22:44:18 -0400 Bryan Bush <bbushvt@gmail.com> wrote:
>
>> I hope this is the right place to ask this question.  I have at 8
>> drive RAID 6 array that I wanted to grow to 13 drives (adding 5 more).
>>  I issued the mdadm command and checked /proc/mdstat and all looked
>> well.  However at some point in time a disk failed and that hung my
>> system.  Upon reboot the array is inactive and I can't get it to
>> reassemble.

> Hmmm... what do you have in /etc/mdadm.conf??

root@diamond:~# cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This file was auto-generated on Fri, 14 Jan 2011 22:22:14 -0500
# by mkconf $Id$

ARRAY /dev/md1 level=raid6 num-devices=8 metadata=01.02 name=1
UUID=fa32e2c5:e7bda20b:32af7c90:c7ee61eb
ARRAY /dev/md0 level=raid5 num-devices=4 metadata=00.90
UUID=989c3b8b:bd60d243:683690d9:73a11dfa

root@diamond:~#

>
> Though if the names have changed at all it would be safer to do
>
>  mdadm -Asf /dev/md1 -u fa32e2c5:e7bda20b:32af7c90:c7ee61eb
>

root@diamond:~# mdadm -Asf /dev/md1 -u fa32e2c5:e7bda20b:32af7c90:c7ee61eb
mdadm: metadata format 01.02 unknown, ignored.
mdadm: metadata format 00.90 unknown, ignored.
mdadm: no devices found for /dev/md1
root@diamond:~#

Didn't seem to help.
Any other ideas?
Thanks
-Bryan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID6 grow failed

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2487 bytes --]

On Wed, 28 Mar 2012 03:54:45 -0400 Bryan Bush <bbushvt@gmail.com> wrote:

> On Wed, Mar 28, 2012 at 12:18 AM, NeilBrown <neilb@suse.de> wrote:
> > On Tue, 27 Mar 2012 22:44:18 -0400 Bryan Bush <bbushvt@gmail.com> wrote:
> >
> >> I hope this is the right place to ask this question.  I have at 8
> >> drive RAID 6 array that I wanted to grow to 13 drives (adding 5 more).
> >>  I issued the mdadm command and checked /proc/mdstat and all looked
> >> well.  However at some point in time a disk failed and that hung my
> >> system.  Upon reboot the array is inactive and I can't get it to
> >> reassemble.
> 
> > Hmmm... what do you have in /etc/mdadm.conf??
> 
> root@diamond:~# cat /etc/mdadm/mdadm.conf
> # mdadm.conf
> #
> # Please refer to mdadm.conf(5) for information about this file.
> #
> 
> # by default, scan all partitions (/proc/partitions) for MD superblocks.
> # alternatively, specify devices to scan, using wildcards if desired.
> DEVICE partitions
> 
> # auto-create devices with Debian standard permissions
> CREATE owner=root group=disk mode=0660 auto=yes
> 
> # automatically tag new arrays as belonging to the local system
> HOMEHOST <system>
> 
> # instruct the monitoring daemon where to send mail alerts
> MAILADDR root
> 
> # definitions of existing MD arrays
> 
> # This file was auto-generated on Fri, 14 Jan 2011 22:22:14 -0500
> # by mkconf $Id$
> 
> ARRAY /dev/md1 level=raid6 num-devices=8 metadata=01.02 name=1
> UUID=fa32e2c5:e7bda20b:32af7c90:c7ee61eb

I think this array now has 13 devices, not 8.  You might want to fix that.
If you do, it might all magically start working.

(also get rid of the metadata= or make it metadata=1.2)

Or just get rid of the num-devices= bit.

> ARRAY /dev/md0 level=raid5 num-devices=4 metadata=00.90
> UUID=989c3b8b:bd60d243:683690d9:73a11dfa
> 
> root@diamond:~#
> 
> >
> > Though if the names have changed at all it would be safer to do
> >
> >  mdadm -Asf /dev/md1 -u fa32e2c5:e7bda20b:32af7c90:c7ee61eb
> >
> 
> root@diamond:~# mdadm -Asf /dev/md1 -u fa32e2c5:e7bda20b:32af7c90:c7ee61eb
> mdadm: metadata format 01.02 unknown, ignored.
> mdadm: metadata format 00.90 unknown, ignored.
> mdadm: no devices found for /dev/md1

Odd.  I the above change to mdadm.conf doesn't help, try rerunning this
command with --verbose and report the result.

NeilBrown


> root@diamond:~#
> 
> Didn't seem to help.
> Any other ideas?
> Thanks
> -Bryan


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID6 grow failed

[Bryan Bush]

Changing the number of devices to 12 worked!
Thanks!
-Bryan

On Wed, Mar 28, 2012 at 4:26 AM, NeilBrown <neilb@suse.de> wrote:
> On Wed, 28 Mar 2012 03:54:45 -0400 Bryan Bush <bbushvt@gmail.com> wrote:
>
>> On Wed, Mar 28, 2012 at 12:18 AM, NeilBrown <neilb@suse.de> wrote:
>> > On Tue, 27 Mar 2012 22:44:18 -0400 Bryan Bush <bbushvt@gmail.com> wrote:
>> >
>> >> I hope this is the right place to ask this question.  I have at 8
>> >> drive RAID 6 array that I wanted to grow to 13 drives (adding 5 more).
>> >>  I issued the mdadm command and checked /proc/mdstat and all looked
>> >> well.  However at some point in time a disk failed and that hung my
>> >> system.  Upon reboot the array is inactive and I can't get it to
>> >> reassemble.
>>
>> > Hmmm... what do you have in /etc/mdadm.conf??
>>
>> root@diamond:~# cat /etc/mdadm/mdadm.conf
>> # mdadm.conf
>> #
>> # Please refer to mdadm.conf(5) for information about this file.
>> #
>>
>> # by default, scan all partitions (/proc/partitions) for MD superblocks.
>> # alternatively, specify devices to scan, using wildcards if desired.
>> DEVICE partitions
>>
>> # auto-create devices with Debian standard permissions
>> CREATE owner=root group=disk mode=0660 auto=yes
>>
>> # automatically tag new arrays as belonging to the local system
>> HOMEHOST <system>
>>
>> # instruct the monitoring daemon where to send mail alerts
>> MAILADDR root
>>
>> # definitions of existing MD arrays
>>
>> # This file was auto-generated on Fri, 14 Jan 2011 22:22:14 -0500
>> # by mkconf $Id$
>>
>> ARRAY /dev/md1 level=raid6 num-devices=8 metadata=01.02 name=1
>> UUID=fa32e2c5:e7bda20b:32af7c90:c7ee61eb
>
> I think this array now has 13 devices, not 8.  You might want to fix that.
> If you do, it might all magically start working.
>
> (also get rid of the metadata= or make it metadata=1.2)
>
> Or just get rid of the num-devices= bit.
>
>> ARRAY /dev/md0 level=raid5 num-devices=4 metadata=00.90
>> UUID=989c3b8b:bd60d243:683690d9:73a11dfa
>>
>> root@diamond:~#
>>
>> >
>> > Though if the names have changed at all it would be safer to do
>> >
>> >  mdadm -Asf /dev/md1 -u fa32e2c5:e7bda20b:32af7c90:c7ee61eb
>> >
>>
>> root@diamond:~# mdadm -Asf /dev/md1 -u fa32e2c5:e7bda20b:32af7c90:c7ee61eb
>> mdadm: metadata format 01.02 unknown, ignored.
>> mdadm: metadata format 00.90 unknown, ignored.
>> mdadm: no devices found for /dev/md1
>
> Odd.  I the above change to mdadm.conf doesn't help, try rerunning this
> command with --verbose and report the result.
>
> NeilBrown
>
>
>> root@diamond:~#
>>
>> Didn't seem to help.
>> Any other ideas?
>> Thanks
>> -Bryan
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID6 grow failed

[Bryan Bush]

I've had another md issue.  Not sure exactly what caused it but my
array is offline again.  The reshape completed and I was about to
insert a new drive to replace the one that failed on the initial
reshape.
/proc/mdstat contains
md1 : inactive sdk1[13](S) sdh1[9](S) sdi1[10](S) sdc1[8](S)
sdj1[11](S) sdb1[4](S) sdd1[6](S) sde1[7](S) sdl1[0](S) sdn1[2](S)
sdo1[3](S)
      21488647746 blocks super 1.2

"mdadm -As --verbose /dev/md1 -u fa32e2c5:e7bda20b:32af7c90:c7ee61eb" returns
mdadm: looking for devices for /dev/md1
<cut>
mdadm: /dev/sdo1 is identified as a member of /dev/md1, slot 3.
mdadm: /dev/sdn1 is identified as a member of /dev/md1, slot 2.
mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 10.
mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 0.
mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 8.
mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 11.
mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 12.
mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 7.
mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 6.
mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 5.
mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 4.
mdadm: Cannot open /dev/sdl1: Device or resource busy

Those are the 11 surviving disks from my 13 disk raid 6.

mdadm -E on those disks gives me
root@diamond:/# mdadm -E /dev/sd[onjlkihedcb]1
mdadm: metadata format 00.90 unknown, ignored.
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : b7c75aff:703e77dd:8be71623:24cfd5c4

    Update Time : Sat Mar 31 02:57:09 2012
       Checksum : 2aa46cd7 - correct
         Events : 31205

     Chunk Size : 64K

    Array Slot : 4 (0, failed, 2, 3, 4, failed, 6, 7, 5, failed,
failed, failed, failed, failed)
   Array State : u_uuUuuu_____ 7 failed
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 73f50290:c36a95a1:99de8ad3:2a4e84c8

    Update Time : Sat Mar 31 02:57:09 2012
       Checksum : 338d3cfe - correct
         Events : 31205

     Chunk Size : 64K

    Array Slot : 8 (0, failed, 2, 3, 4, failed, 6, 7, 5, failed,
failed, failed, failed, failed)
   Array State : u_uuuUuu_____ 7 failed
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : de927013:298f7906:5abdae1e:ccab9188

    Update Time : Sat Mar 31 02:57:09 2012
       Checksum : 27103b2e - correct
         Events : 31205

     Chunk Size : 64K

    Array Slot : 6 (0, failed, 2, 3, 4, failed, 6, 7, 5, failed,
failed, failed, failed, failed)
   Array State : u_uuuuUu_____ 7 failed
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : a258e2c3:e4abf94c:33be105e:fff62610

    Update Time : Sat Mar 31 02:57:09 2012
       Checksum : e4f969ba - correct
         Events : 31205

     Chunk Size : 64K

    Array Slot : 7 (0, failed, 2, 3, 4, failed, 6, 7, 5, failed,
failed, failed, failed, failed)
   Array State : u_uuuuuU_____ 7 failed
/dev/sdh1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : bf54f25f:3a82eb7e:4b852f91:6857c476

    Update Time : Sat Mar 31 02:54:55 2012
       Checksum : 4cd96340 - correct
         Events : 31205

     Chunk Size : 64K

    Array Slot : 9 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_uuuuuuuuuuU 2 failed
/dev/sdi1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : e239f08a:e56a2abd:18bd6e8d:6e614aa2

    Update Time : Sat Mar 31 02:54:55 2012
       Checksum : dddb72e2 - correct
         Events : 31205

     Chunk Size : 64K

    Array Slot : 10 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_uuuuuuuuuUu 2 failed
/dev/sdj1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : f04f830d:ffc3352f:443c92c0:b03ec412

    Update Time : Sat Mar 31 02:54:55 2012
       Checksum : 76173e74 - correct
         Events : 31201

     Chunk Size : 64K

    Array Slot : 11 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_uuuuuuuuUuu 2 failed
/dev/sdk1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 74bb612c:34c2ddf6:4828ed5c:22ea7b38

    Update Time : Sat Mar 31 02:54:55 2012
       Checksum : 1eb03faa - correct
         Events : 31205

     Chunk Size : 64K

    Array Slot : 13 (0, failed, 2, 3, 4, failed, 6, 7, 5, 12, 11, 10, 9, 8)
   Array State : u_uuuuuuUuuuu 2 failed
/dev/sdl1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 14c53549:bd43ec01:794d70ff:3ea30a9d

    Update Time : Sat Mar 31 02:57:09 2012
       Checksum : 4d82a984 - correct
         Events : 31205

     Chunk Size : 64K

    Array Slot : 0 (0, failed, 2, 3, 4, failed, 6, 7, 5, failed,
failed, failed, failed, failed)
   Array State : U_uuuuuu_____ 7 failed
/dev/sdn1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : dd08d581:c542403f:33ab15bb:e5a7f122

    Update Time : Sat Mar 31 02:57:09 2012
       Checksum : 5024eb8 - correct
         Events : 31205

     Chunk Size : 64K

    Array Slot : 2 (0, failed, 2, 3, 4, failed, 6, 7, 5, failed,
failed, failed, failed, failed)
   Array State : u_Uuuuuu_____ 7 failed
/dev/sdo1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : fa32e2c5:e7bda20b:32af7c90:c7ee61eb
           Name : 1
  Creation Time : Sat Sep 18 17:05:52 2010
     Raid Level : raid6
   Raid Devices : 13

 Avail Dev Size : 3907026863 (1863.02 GiB 2000.40 GB)
     Array Size : 42977294976 (20493.17 GiB 22004.38 GB)
  Used Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
    Data Offset : 272 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 6efe22ba:87cc1ffb:a070dbd0:b9020b5e

    Update Time : Sat Mar 31 02:57:09 2012
       Checksum : 4a0eee4e - correct
         Events : 31205

     Chunk Size : 64K

    Array Slot : 3 (0, failed, 2, 3, 4, failed, 6, 7, 5, failed,
failed, failed, failed, failed)
   Array State : u_uUuuuu_____ 7 failed
root@diamond:/#

and my mdadm.conf contains
ARRAY /dev/md1 level=raid6 num-devices=13 name=1
UUID=fa32e2c5:e7bda20b:32af7c90:c7ee61eb

I'm confused as to why /dev/sdl1 says device or resource is busy..
Any ideas?
Thanks
-Bryan




On Wed, Mar 28, 2012 at 6:35 AM, Bryan Bush <bbushvt@gmail.com> wrote:
> Changing the number of devices to 12 worked!
> Thanks!
> -Bryan
>
> On Wed, Mar 28, 2012 at 4:26 AM, NeilBrown <neilb@suse.de> wrote:
>> On Wed, 28 Mar 2012 03:54:45 -0400 Bryan Bush <bbushvt@gmail.com> wrote:
>>
>>> On Wed, Mar 28, 2012 at 12:18 AM, NeilBrown <neilb@suse.de> wrote:
>>> > On Tue, 27 Mar 2012 22:44:18 -0400 Bryan Bush <bbushvt@gmail.com> wrote:
>>> >
>>> >> I hope this is the right place to ask this question.  I have at 8
>>> >> drive RAID 6 array that I wanted to grow to 13 drives (adding 5 more).
>>> >>  I issued the mdadm command and checked /proc/mdstat and all looked
>>> >> well.  However at some point in time a disk failed and that hung my
>>> >> system.  Upon reboot the array is inactive and I can't get it to
>>> >> reassemble.
>>>
>>> > Hmmm... what do you have in /etc/mdadm.conf??
>>>
>>> root@diamond:~# cat /etc/mdadm/mdadm.conf
>>> # mdadm.conf
>>> #
>>> # Please refer to mdadm.conf(5) for information about this file.
>>> #
>>>
>>> # by default, scan all partitions (/proc/partitions) for MD superblocks.
>>> # alternatively, specify devices to scan, using wildcards if desired.
>>> DEVICE partitions
>>>
>>> # auto-create devices with Debian standard permissions
>>> CREATE owner=root group=disk mode=0660 auto=yes
>>>
>>> # automatically tag new arrays as belonging to the local system
>>> HOMEHOST <system>
>>>
>>> # instruct the monitoring daemon where to send mail alerts
>>> MAILADDR root
>>>
>>> # definitions of existing MD arrays
>>>
>>> # This file was auto-generated on Fri, 14 Jan 2011 22:22:14 -0500
>>> # by mkconf $Id$
>>>
>>> ARRAY /dev/md1 level=raid6 num-devices=8 metadata=01.02 name=1
>>> UUID=fa32e2c5:e7bda20b:32af7c90:c7ee61eb
>>
>> I think this array now has 13 devices, not 8.  You might want to fix that.
>> If you do, it might all magically start working.
>>
>> (also get rid of the metadata= or make it metadata=1.2)
>>
>> Or just get rid of the num-devices= bit.
>>
>>> ARRAY /dev/md0 level=raid5 num-devices=4 metadata=00.90
>>> UUID=989c3b8b:bd60d243:683690d9:73a11dfa
>>>
>>> root@diamond:~#
>>>
>>> >
>>> > Though if the names have changed at all it would be safer to do
>>> >
>>> >  mdadm -Asf /dev/md1 -u fa32e2c5:e7bda20b:32af7c90:c7ee61eb
>>> >
>>>
>>> root@diamond:~# mdadm -Asf /dev/md1 -u fa32e2c5:e7bda20b:32af7c90:c7ee61eb
>>> mdadm: metadata format 01.02 unknown, ignored.
>>> mdadm: metadata format 00.90 unknown, ignored.
>>> mdadm: no devices found for /dev/md1
>>
>> Odd.  I the above change to mdadm.conf doesn't help, try rerunning this
>> command with --verbose and report the result.
>>
>> NeilBrown
>>
>>
>>> root@diamond:~#
>>>
>>> Didn't seem to help.
>>> Any other ideas?
>>> Thanks
>>> -Bryan
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID6 grow failed

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1572 bytes --]

On Sat, 31 Mar 2012 15:38:23 -0400 Bryan Bush <bbushvt@gmail.com> wrote:

> I've had another md issue.  Not sure exactly what caused it but my
> array is offline again.  The reshape completed and I was about to
> insert a new drive to replace the one that failed on the initial
> reshape.
> /proc/mdstat contains
> md1 : inactive sdk1[13](S) sdh1[9](S) sdi1[10](S) sdc1[8](S)
> sdj1[11](S) sdb1[4](S) sdd1[6](S) sde1[7](S) sdl1[0](S) sdn1[2](S)
> sdo1[3](S)
>       21488647746 blocks super 1.2
> 
> "mdadm -As --verbose /dev/md1 -u fa32e2c5:e7bda20b:32af7c90:c7ee61eb" returns
> mdadm: looking for devices for /dev/md1
> <cut>
> mdadm: /dev/sdo1 is identified as a member of /dev/md1, slot 3.
> mdadm: /dev/sdn1 is identified as a member of /dev/md1, slot 2.
> mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 10.
> mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 0.
> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 8.
> mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 11.
> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 12.
> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 7.
> mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 6.
> mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 5.
> mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 4.
> mdadm: Cannot open /dev/sdl1: Device or resource busy

Stop the array first. - it is half-started which confused mdadm.
i.e
   mdadm -S /dev/md1
   mdadm -As .......

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID6 grow failed

[Bryan Bush]

I still get the same error
mdadm: Cannot open /dev/sdl1: Device or resource busy
Thanks
-Bryan


On Sat, Mar 31, 2012 at 6:00 PM, NeilBrown <neilb@suse.de> wrote:
> On Sat, 31 Mar 2012 15:38:23 -0400 Bryan Bush <bbushvt@gmail.com> wrote:
>
>> I've had another md issue.  Not sure exactly what caused it but my
>> array is offline again.  The reshape completed and I was about to
>> insert a new drive to replace the one that failed on the initial
>> reshape.
>> /proc/mdstat contains
>> md1 : inactive sdk1[13](S) sdh1[9](S) sdi1[10](S) sdc1[8](S)
>> sdj1[11](S) sdb1[4](S) sdd1[6](S) sde1[7](S) sdl1[0](S) sdn1[2](S)
>> sdo1[3](S)
>>       21488647746 blocks super 1.2
>>
>> "mdadm -As --verbose /dev/md1 -u fa32e2c5:e7bda20b:32af7c90:c7ee61eb" returns
>> mdadm: looking for devices for /dev/md1
>> <cut>
>> mdadm: /dev/sdo1 is identified as a member of /dev/md1, slot 3.
>> mdadm: /dev/sdn1 is identified as a member of /dev/md1, slot 2.
>> mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 10.
>> mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 0.
>> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 8.
>> mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 11.
>> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 12.
>> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 7.
>> mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 6.
>> mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 5.
>> mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 4.
>> mdadm: Cannot open /dev/sdl1: Device or resource busy
>
> Stop the array first. - it is half-started which confused mdadm.
> i.e
>   mdadm -S /dev/md1
>   mdadm -As .......
>
> NeilBrown
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID6 grow failed

[Bryan Bush]

Additional info that might be useful.  When I specify which disks to
use in the array, I get this
root@diamond:/# mdadm -A --verbose /dev/md1 /dev/sd[onjlkuhedcb]1
mdadm: looking for devices for /dev/md1
mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 4.
mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 5.
mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 6.
mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 7.
mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 12.
mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 10.
mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 8.
mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 0.
mdadm: /dev/sdn1 is identified as a member of /dev/md1, slot 2.
mdadm: /dev/sdo1 is identified as a member of /dev/md1, slot 3.
mdadm: device 8 in /dev/md1 has wrong state in superblock, but
/dev/sdk1 seems ok
mdadm: device 10 in /dev/md1 has wrong state in superblock, but
/dev/sdj1 seems ok
mdadm: device 12 in /dev/md1 has wrong state in superblock, but
/dev/sdh1 seems ok
mdadm: SET_ARRAY_INFO failed for /dev/md1: Device or resource busy
root@diamond:/#

Thanks
-Bryan


On Sat, Mar 31, 2012 at 7:24 PM, Bryan Bush <bbushvt@gmail.com> wrote:
> I still get the same error
> mdadm: Cannot open /dev/sdl1: Device or resource busy
> Thanks
> -Bryan
>
>
> On Sat, Mar 31, 2012 at 6:00 PM, NeilBrown <neilb@suse.de> wrote:
>> On Sat, 31 Mar 2012 15:38:23 -0400 Bryan Bush <bbushvt@gmail.com> wrote:
>>
>>> I've had another md issue.  Not sure exactly what caused it but my
>>> array is offline again.  The reshape completed and I was about to
>>> insert a new drive to replace the one that failed on the initial
>>> reshape.
>>> /proc/mdstat contains
>>> md1 : inactive sdk1[13](S) sdh1[9](S) sdi1[10](S) sdc1[8](S)
>>> sdj1[11](S) sdb1[4](S) sdd1[6](S) sde1[7](S) sdl1[0](S) sdn1[2](S)
>>> sdo1[3](S)
>>>       21488647746 blocks super 1.2
>>>
>>> "mdadm -As --verbose /dev/md1 -u fa32e2c5:e7bda20b:32af7c90:c7ee61eb" returns
>>> mdadm: looking for devices for /dev/md1
>>> <cut>
>>> mdadm: /dev/sdo1 is identified as a member of /dev/md1, slot 3.
>>> mdadm: /dev/sdn1 is identified as a member of /dev/md1, slot 2.
>>> mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 10.
>>> mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 0.
>>> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 8.
>>> mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 11.
>>> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 12.
>>> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 7.
>>> mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 6.
>>> mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 5.
>>> mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 4.
>>> mdadm: Cannot open /dev/sdl1: Device or resource busy
>>
>> Stop the array first. - it is half-started which confused mdadm.
>> i.e
>>   mdadm -S /dev/md1
>>   mdadm -As .......
>>
>> NeilBrown
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID6 grow failed

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1556 bytes --]

On Sun, 1 Apr 2012 03:05:29 -0400 Bryan Bush <bbushvt@gmail.com> wrote:

> Additional info that might be useful.  When I specify which disks to
> use in the array, I get this
> root@diamond:/# mdadm -A --verbose /dev/md1 /dev/sd[onjlkuhedcb]1
> mdadm: looking for devices for /dev/md1
> mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 4.
> mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 5.
> mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 6.
> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 7.
> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 12.
> mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 10.
> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 8.
> mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 0.
> mdadm: /dev/sdn1 is identified as a member of /dev/md1, slot 2.
> mdadm: /dev/sdo1 is identified as a member of /dev/md1, slot 3.
> mdadm: device 8 in /dev/md1 has wrong state in superblock, but
> /dev/sdk1 seems ok
> mdadm: device 10 in /dev/md1 has wrong state in superblock, but
> /dev/sdj1 seems ok
> mdadm: device 12 in /dev/md1 has wrong state in superblock, but
> /dev/sdh1 seems ok
> mdadm: SET_ARRAY_INFO failed for /dev/md1: Device or resource busy
> root@diamond:/#

1/ Are you absolutely certain you did
     mdadm -S /dev/md1
 first?  Because it really looks like you didn't.

2/ Any kernel log messages?
    dmesg | tail -n 50
  immediately after the failed "mdadm -A" command.

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID6 grow failed

[Bryan Bush]

root@diamond:/# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md1 : inactive sdk1[13](S) sdj1[11](S) sdh1[9](S) sdo1[3](S)
sdd1[6](S) sde1[7](S) sdn1[2](S) sdl1[0](S) sdb1[4](S) sdc1[8](S)
      19535134315 blocks super 1.2

md0 : active raid5 sdg1[3] sda1[0] sdf1[1] sdq1[2]
      2929686528 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>
root@diamond:/# mdadm -S /dev/md1
mdadm: stopped /dev/md1
root@diamond:/# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : active raid5 sdg1[3] sda1[0] sdf1[1] sdq1[2]
      2929686528 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>
root@diamond:/# mdadm -A --verbose /dev/md1 /dev/sd[onjlkuhedcb]1
mdadm: looking for devices for /dev/md1
mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 4.
mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 5.
mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 6.
mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 7.
mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 12.
mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 10.
mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 8.
mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 0.
mdadm: /dev/sdn1 is identified as a member of /dev/md1, slot 2.
mdadm: /dev/sdo1 is identified as a member of /dev/md1, slot 3.
mdadm: device 8 in /dev/md1 has wrong state in superblock, but
/dev/sdk1 seems ok
mdadm: device 10 in /dev/md1 has wrong state in superblock, but
/dev/sdj1 seems ok
mdadm: device 12 in /dev/md1 has wrong state in superblock, but
/dev/sdh1 seems ok
mdadm: SET_ARRAY_INFO failed for /dev/md1: Device or resource busy
root@diamond:/# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md1 : inactive sdk1[13](S) sdj1[11](S) sdd1[6](S) sdh1[9](S)
sdo1[3](S) sde1[7](S) sdn1[2](S) sdl1[0](S) sdc1[8](S) sdb1[4](S)
      19535134315 blocks super 1.2

md0 : active raid5 sdg1[3] sda1[0] sdf1[1] sdq1[2]
      2929686528 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>


Output from /var/log/messages for the mdadm -A
Apr  1 20:00:57 diamond kernel: [106978.432900] md: md1 stopped.
Apr  1 20:00:57 diamond kernel: [106978.493151] md: bind<sdc1>
Apr  1 20:00:57 diamond kernel: [106978.494551] md: bind<sdb1>
Apr  1 20:00:57 diamond kernel: [106978.496256] md: bind<sdd1>
Apr  1 20:00:57 diamond kernel: [106978.516939] md: array md1 already has disks!
Apr  1 20:00:57 diamond kernel: [106978.525475] md: bind<sdo1>
Apr  1 20:00:57 diamond kernel: [106978.527509] md: bind<sdn1>
Apr  1 20:00:58 diamond kernel: [106978.694915] md: bind<sde1>
Apr  1 20:00:58 diamond kernel: [106978.726157] md: bind<sdl1>
Apr  1 20:00:58 diamond kernel: [106978.868815] md: bind<sdh1>
Apr  1 20:00:58 diamond kernel: [106979.079319] md: bind<sdj1>
Apr  1 20:00:58 diamond kernel: [106979.280302] md: bind<sdk1>


Thanks
-Bryan



On Sun, Apr 1, 2012 at 7:54 PM, NeilBrown <neilb@suse.de> wrote:
> On Sun, 1 Apr 2012 03:05:29 -0400 Bryan Bush <bbushvt@gmail.com> wrote:
>
>> Additional info that might be useful.  When I specify which disks to
>> use in the array, I get this
>> root@diamond:/# mdadm -A --verbose /dev/md1 /dev/sd[onjlkuhedcb]1
>> mdadm: looking for devices for /dev/md1
>> mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 4.
>> mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 5.
>> mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 6.
>> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 7.
>> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 12.
>> mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 10.
>> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 8.
>> mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 0.
>> mdadm: /dev/sdn1 is identified as a member of /dev/md1, slot 2.
>> mdadm: /dev/sdo1 is identified as a member of /dev/md1, slot 3.
>> mdadm: device 8 in /dev/md1 has wrong state in superblock, but
>> /dev/sdk1 seems ok
>> mdadm: device 10 in /dev/md1 has wrong state in superblock, but
>> /dev/sdj1 seems ok
>> mdadm: device 12 in /dev/md1 has wrong state in superblock, but
>> /dev/sdh1 seems ok
>> mdadm: SET_ARRAY_INFO failed for /dev/md1: Device or resource busy
>> root@diamond:/#
>
> 1/ Are you absolutely certain you did
>     mdadm -S /dev/md1
>  first?  Because it really looks like you didn't.
>
> 2/ Any kernel log messages?
>    dmesg | tail -n 50
>  immediately after the failed "mdadm -A" command.
>
> NeilBrown
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID6 grow failed

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3438 bytes --]

On Sun, 1 Apr 2012 20:02:57 -0400 Bryan Bush <bbushvt@gmail.com> wrote:

> root@diamond:/# cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md1 : inactive sdk1[13](S) sdj1[11](S) sdh1[9](S) sdo1[3](S)
> sdd1[6](S) sde1[7](S) sdn1[2](S) sdl1[0](S) sdb1[4](S) sdc1[8](S)
>       19535134315 blocks super 1.2
> 
> md0 : active raid5 sdg1[3] sda1[0] sdf1[1] sdq1[2]
>       2929686528 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
> 
> unused devices: <none>
> root@diamond:/# mdadm -S /dev/md1
> mdadm: stopped /dev/md1
> root@diamond:/# cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md0 : active raid5 sdg1[3] sda1[0] sdf1[1] sdq1[2]
>       2929686528 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
> 
> unused devices: <none>
> root@diamond:/# mdadm -A --verbose /dev/md1 /dev/sd[onjlkuhedcb]1
> mdadm: looking for devices for /dev/md1
> mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 4.
> mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 5.
> mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 6.
> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 7.
> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 12.
> mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 10.
> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 8.
> mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 0.
> mdadm: /dev/sdn1 is identified as a member of /dev/md1, slot 2.
> mdadm: /dev/sdo1 is identified as a member of /dev/md1, slot 3.
> mdadm: device 8 in /dev/md1 has wrong state in superblock, but
> /dev/sdk1 seems ok
> mdadm: device 10 in /dev/md1 has wrong state in superblock, but
> /dev/sdj1 seems ok
> mdadm: device 12 in /dev/md1 has wrong state in superblock, but
> /dev/sdh1 seems ok
> mdadm: SET_ARRAY_INFO failed for /dev/md1: Device or resource busy
> root@diamond:/# cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md1 : inactive sdk1[13](S) sdj1[11](S) sdd1[6](S) sdh1[9](S)
> sdo1[3](S) sde1[7](S) sdn1[2](S) sdl1[0](S) sdc1[8](S) sdb1[4](S)
>       19535134315 blocks super 1.2
> 
> md0 : active raid5 sdg1[3] sda1[0] sdf1[1] sdq1[2]
>       2929686528 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
> 
> unused devices: <none>
> 
> 
> Output from /var/log/messages for the mdadm -A
> Apr  1 20:00:57 diamond kernel: [106978.432900] md: md1 stopped.
> Apr  1 20:00:57 diamond kernel: [106978.493151] md: bind<sdc1>
> Apr  1 20:00:57 diamond kernel: [106978.494551] md: bind<sdb1>
> Apr  1 20:00:57 diamond kernel: [106978.496256] md: bind<sdd1>
> Apr  1 20:00:57 diamond kernel: [106978.516939] md: array md1 already has disks!

That is where SET_ARRAY_INFO is failing ... but why does md1 already have
disks I wonder...

either mdadm has some weird bug - what version are you running???

or something else is messing with md1.

Maybe udev is noticing those devices again for some reason and trying to add
them to the array independently.
You could run
   udevadm monitor

at the same time and see what happens.
Also look in /lib/udev/rules.d or /etc/udev/rules.d to find an entry that 
run "mdadm -I" or "mdadm --incremental" and  try commenting that entry out.

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID6 grow failed

[Bryan Bush]

my mdadm version is
root@diamond:/# mdadm -V
mdadm - v2.6.7.1 - 15th October 2008

Here is the output from udevadm monitor
KERNEL[1333327039.975612] add      /devices/virtual/block/md1 (block)
KERNEL[1333327039.975652] add      /devices/virtual/bdi/9:1 (bdi)
KERNEL[1333327039.975748] change   /devices/virtual/block/md1 (block)
UDEV  [1333327039.975859] add      /devices/virtual/block/md1 (block)
UDEV  [1333327039.975889] add      /devices/virtual/bdi/9:1 (bdi)
UDEV  [1333327039.976131] change   /devices/virtual/block/md1 (block)
KERNEL[1333327040.022682] change
/devices/pci0000:00/0000:00:13.0/0000:03:00.0/host2/target2:0:0/2:0:0:0/block/sdb/sdb1
(block)
KERNEL[1333327040.023186] change
/devices/pci0000:00/0000:00:13.0/0000:03:00.0/host2/target2:1:0/2:1:0:0/block/sdc/sdc1
(block)
KERNEL[1333327040.023765] change
/devices/pci0000:00/0000:00:13.0/0000:03:00.0/host2/target2:2:0/2:2:0:0/block/sdd/sdd1
(block)
KERNEL[1333327040.023969] change
/devices/pci0000:00/0000:00:13.0/0000:03:00.0/host2/target2:3:0/2:3:0:0/block/sde/sde1
(block)
UDEV  [1333327040.033204] change
/devices/pci0000:00/0000:00:13.0/0000:03:00.0/host2/target2:1:0/2:1:0:0/block/sdc/sdc1
(block)
KERNEL[1333327040.034437] change
/devices/pci0000:00/0000:00:16.0/0000:04:00.0/host7/target7:0:0/7:0:0:0/block/sdh/sdh1
(block)
UDEV  [1333327040.035269] change
/devices/pci0000:00/0000:00:13.0/0000:03:00.0/host2/target2:0:0/2:0:0:0/block/sdb/sdb1
(block)
KERNEL[1333327040.044692] change
/devices/pci0000:00/0000:00:16.0/0000:04:00.0/host12/target12:0:0/12:0:0:0/block/sdj/sdj1
(block)
KERNEL[1333327040.057077] change
/devices/pci0000:00/0000:00:16.0/0000:04:00.0/host13/target13:0:0/13:0:0:0/block/sdk/sdk1
(block)
KERNEL[1333327040.057430] change
/devices/pci0000:00/0000:00:18.0/0000:06:00.0/host17/target17:0:0/17:0:0:0/block/sdl/sdl1
(block)
KERNEL[1333327040.057658] change
/devices/pci0000:00/0000:00:18.0/0000:06:00.0/host17/target17:2:0/17:2:0:0/block/sdn/sdn1
(block)
KERNEL[1333327040.057888] change
/devices/pci0000:00/0000:00:18.0/0000:06:00.0/host17/target17:3:0/17:3:0:0/block/sdo/sdo1
(block)
UDEV  [1333327040.067716] change
/devices/pci0000:00/0000:00:18.0/0000:06:00.0/host17/target17:0:0/17:0:0:0/block/sdl/sdl1
(block)
UDEV  [1333327040.235203] change
/devices/pci0000:00/0000:00:13.0/0000:03:00.0/host2/target2:3:0/2:3:0:0/block/sde/sde1
(block)
UDEV  [1333327040.269335] change
/devices/pci0000:00/0000:00:18.0/0000:06:00.0/host17/target17:3:0/17:3:0:0/block/sdo/sdo1
(block)
UDEV  [1333327040.422753] change
/devices/pci0000:00/0000:00:16.0/0000:04:00.0/host7/target7:0:0/7:0:0:0/block/sdh/sdh1
(block)
UDEV  [1333327040.457015] change
/devices/pci0000:00/0000:00:13.0/0000:03:00.0/host2/target2:2:0/2:2:0:0/block/sdd/sdd1
(block)
UDEV  [1333327040.480483] change
/devices/pci0000:00/0000:00:18.0/0000:06:00.0/host17/target17:2:0/17:2:0:0/block/sdn/sdn1
(block)
UDEV  [1333327040.633694] change
/devices/pci0000:00/0000:00:16.0/0000:04:00.0/host12/target12:0:0/12:0:0:0/block/sdj/sdj1
(block)
UDEV  [1333327040.845071] change
/devices/pci0000:00/0000:00:16.0/0000:04:00.0/host13/target13:0:0/13:0:0:0/block/sdk/sdk1
(block)

When I use mdadm 3.2.1 I get
root@diamond:~/mdadm/mdadm-3.2.1# ./mdadm -A --verbose /dev/md1
/dev/sd[onjlkuhedcb]1
mdadm: looking for devices for /dev/md1
mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 4.
mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 5.
mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 6.
mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 7.
mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 12.
mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 10.
mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 8.
mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 0.
mdadm: /dev/sdn1 is identified as a member of /dev/md1, slot 2.
mdadm: /dev/sdo1 is identified as a member of /dev/md1, slot 3.
mdadm: device 8 in /dev/md1 has wrong state in superblock, but
/dev/sdk1 seems ok
mdadm: device 10 in /dev/md1 has wrong state in superblock, but
/dev/sdj1 seems ok
mdadm: device 12 in /dev/md1 has wrong state in superblock, but
/dev/sdh1 seems ok
mdadm: no uptodate device for slot 1 of /dev/md1
mdadm: added /dev/sdn1 to /dev/md1 as 2
mdadm: added /dev/sdo1 to /dev/md1 as 3
mdadm: added /dev/sdb1 to /dev/md1 as 4
mdadm: added /dev/sdc1 to /dev/md1 as 5
mdadm: added /dev/sdd1 to /dev/md1 as 6
mdadm: added /dev/sde1 to /dev/md1 as 7
mdadm: added /dev/sdk1 to /dev/md1 as 8
mdadm: no uptodate device for slot 9 of /dev/md1
mdadm: added /dev/sdj1 to /dev/md1 as 10
mdadm: no uptodate device for slot 11 of /dev/md1
mdadm: added /dev/sdh1 to /dev/md1 as 12
mdadm: added /dev/sdl1 to /dev/md1 as 0
mdadm: /dev/md1 assembled from 10 drives - not enough to start the array.


Should I try to force it?  Worried it might make things worse.
Thanks
-Bryan



On Sun, Apr 1, 2012 at 8:26 PM, NeilBrown <neilb@suse.de> wrote:
> On Sun, 1 Apr 2012 20:02:57 -0400 Bryan Bush <bbushvt@gmail.com> wrote:
>
>> root@diamond:/# cat /proc/mdstat
>> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
>> [raid4] [raid10]
>> md1 : inactive sdk1[13](S) sdj1[11](S) sdh1[9](S) sdo1[3](S)
>> sdd1[6](S) sde1[7](S) sdn1[2](S) sdl1[0](S) sdb1[4](S) sdc1[8](S)
>>       19535134315 blocks super 1.2
>>
>> md0 : active raid5 sdg1[3] sda1[0] sdf1[1] sdq1[2]
>>       2929686528 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
>>
>> unused devices: <none>
>> root@diamond:/# mdadm -S /dev/md1
>> mdadm: stopped /dev/md1
>> root@diamond:/# cat /proc/mdstat
>> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
>> [raid4] [raid10]
>> md0 : active raid5 sdg1[3] sda1[0] sdf1[1] sdq1[2]
>>       2929686528 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
>>
>> unused devices: <none>
>> root@diamond:/# mdadm -A --verbose /dev/md1 /dev/sd[onjlkuhedcb]1
>> mdadm: looking for devices for /dev/md1
>> mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 4.
>> mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 5.
>> mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 6.
>> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 7.
>> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 12.
>> mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 10.
>> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 8.
>> mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 0.
>> mdadm: /dev/sdn1 is identified as a member of /dev/md1, slot 2.
>> mdadm: /dev/sdo1 is identified as a member of /dev/md1, slot 3.
>> mdadm: device 8 in /dev/md1 has wrong state in superblock, but
>> /dev/sdk1 seems ok
>> mdadm: device 10 in /dev/md1 has wrong state in superblock, but
>> /dev/sdj1 seems ok
>> mdadm: device 12 in /dev/md1 has wrong state in superblock, but
>> /dev/sdh1 seems ok
>> mdadm: SET_ARRAY_INFO failed for /dev/md1: Device or resource busy
>> root@diamond:/# cat /proc/mdstat
>> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
>> [raid4] [raid10]
>> md1 : inactive sdk1[13](S) sdj1[11](S) sdd1[6](S) sdh1[9](S)
>> sdo1[3](S) sde1[7](S) sdn1[2](S) sdl1[0](S) sdc1[8](S) sdb1[4](S)
>>       19535134315 blocks super 1.2
>>
>> md0 : active raid5 sdg1[3] sda1[0] sdf1[1] sdq1[2]
>>       2929686528 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
>>
>> unused devices: <none>
>>
>>
>> Output from /var/log/messages for the mdadm -A
>> Apr  1 20:00:57 diamond kernel: [106978.432900] md: md1 stopped.
>> Apr  1 20:00:57 diamond kernel: [106978.493151] md: bind<sdc1>
>> Apr  1 20:00:57 diamond kernel: [106978.494551] md: bind<sdb1>
>> Apr  1 20:00:57 diamond kernel: [106978.496256] md: bind<sdd1>
>> Apr  1 20:00:57 diamond kernel: [106978.516939] md: array md1 already has disks!
>
> That is where SET_ARRAY_INFO is failing ... but why does md1 already have
> disks I wonder...
>
> either mdadm has some weird bug - what version are you running???
>
> or something else is messing with md1.
>
> Maybe udev is noticing those devices again for some reason and trying to add
> them to the array independently.
> You could run
>   udevadm monitor
>
> at the same time and see what happens.
> Also look in /lib/udev/rules.d or /etc/udev/rules.d to find an entry that
> run "mdadm -I" or "mdadm --incremental" and  try commenting that entry out.
>
> NeilBrown
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID6 grow failed

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2767 bytes --]

On Sun, 1 Apr 2012 20:44:17 -0400 Bryan Bush <bbushvt@gmail.com> wrote:

> my mdadm version is
> root@diamond:/# mdadm -V
> mdadm - v2.6.7.1 - 15th October 2008

That's rather old... I'm not surprised that it doesn't cope with assembling
arrays that are in the middle of being reshaped.

>> 
> When I use mdadm 3.2.1 I get
> root@diamond:~/mdadm/mdadm-3.2.1# ./mdadm -A --verbose /dev/md1
> /dev/sd[onjlkuhedcb]1
> mdadm: looking for devices for /dev/md1
> mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 4.
> mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 5.
> mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 6.
> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 7.
> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 12.
> mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 10.
> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 8.
> mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 0.
> mdadm: /dev/sdn1 is identified as a member of /dev/md1, slot 2.
> mdadm: /dev/sdo1 is identified as a member of /dev/md1, slot 3.
> mdadm: device 8 in /dev/md1 has wrong state in superblock, but
> /dev/sdk1 seems ok
> mdadm: device 10 in /dev/md1 has wrong state in superblock, but
> /dev/sdj1 seems ok
> mdadm: device 12 in /dev/md1 has wrong state in superblock, but
> /dev/sdh1 seems ok
> mdadm: no uptodate device for slot 1 of /dev/md1
> mdadm: added /dev/sdn1 to /dev/md1 as 2
> mdadm: added /dev/sdo1 to /dev/md1 as 3
> mdadm: added /dev/sdb1 to /dev/md1 as 4
> mdadm: added /dev/sdc1 to /dev/md1 as 5
> mdadm: added /dev/sdd1 to /dev/md1 as 6
> mdadm: added /dev/sde1 to /dev/md1 as 7
> mdadm: added /dev/sdk1 to /dev/md1 as 8
> mdadm: no uptodate device for slot 9 of /dev/md1
> mdadm: added /dev/sdj1 to /dev/md1 as 10
> mdadm: no uptodate device for slot 11 of /dev/md1
> mdadm: added /dev/sdh1 to /dev/md1 as 12
> mdadm: added /dev/sdl1 to /dev/md1 as 0
> mdadm: /dev/md1 assembled from 10 drives - not enough to start the array.

That looks a lot more sensible.
So that array expects 13 drives, but you only have 10.
You'll need to find at least 1 more (preferably 3 more) to have any chance of
success.

> 
> 
> Should I try to force it?  Worried it might make things worse.

force won't help until you find those other devices.

'force' is unlikely to make things worse.  It does the best it can.  The
reason that you need to actually give "--force" (rather than mdadm always
doing the best it case) is that you need to know that something has gone
wrong and so not to trust the contents of the array until you have verified
that your data is safe (most of it will be, but no promises).

NeilBrown



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID6 grow failed

[Bryan Bush]

It looks like my old mdadm (2.6.7.1, ubuntu 10.10) has having issues
and with the newer version (3.2.1) I needed to use force to get it
back up.  Its back up now with 11 of 13 disks, but i've added a
replacement disk, so in about 20 hours I'll be back up to 12 of 13.
Thanks for the help, I've learned a lot on this list and am glad I was
able to find it.
Thanks
-Bryan



On Sun, Apr 1, 2012 at 10:12 PM, NeilBrown <neilb@suse.de> wrote:
> On Sun, 1 Apr 2012 20:44:17 -0400 Bryan Bush <bbushvt@gmail.com> wrote:
>
>> my mdadm version is
>> root@diamond:/# mdadm -V
>> mdadm - v2.6.7.1 - 15th October 2008
>
> That's rather old... I'm not surprised that it doesn't cope with assembling
> arrays that are in the middle of being reshaped.
>
>>>
>> When I use mdadm 3.2.1 I get
>> root@diamond:~/mdadm/mdadm-3.2.1# ./mdadm -A --verbose /dev/md1
>> /dev/sd[onjlkuhedcb]1
>> mdadm: looking for devices for /dev/md1
>> mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 4.
>> mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 5.
>> mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 6.
>> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 7.
>> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 12.
>> mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 10.
>> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 8.
>> mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 0.
>> mdadm: /dev/sdn1 is identified as a member of /dev/md1, slot 2.
>> mdadm: /dev/sdo1 is identified as a member of /dev/md1, slot 3.
>> mdadm: device 8 in /dev/md1 has wrong state in superblock, but
>> /dev/sdk1 seems ok
>> mdadm: device 10 in /dev/md1 has wrong state in superblock, but
>> /dev/sdj1 seems ok
>> mdadm: device 12 in /dev/md1 has wrong state in superblock, but
>> /dev/sdh1 seems ok
>> mdadm: no uptodate device for slot 1 of /dev/md1
>> mdadm: added /dev/sdn1 to /dev/md1 as 2
>> mdadm: added /dev/sdo1 to /dev/md1 as 3
>> mdadm: added /dev/sdb1 to /dev/md1 as 4
>> mdadm: added /dev/sdc1 to /dev/md1 as 5
>> mdadm: added /dev/sdd1 to /dev/md1 as 6
>> mdadm: added /dev/sde1 to /dev/md1 as 7
>> mdadm: added /dev/sdk1 to /dev/md1 as 8
>> mdadm: no uptodate device for slot 9 of /dev/md1
>> mdadm: added /dev/sdj1 to /dev/md1 as 10
>> mdadm: no uptodate device for slot 11 of /dev/md1
>> mdadm: added /dev/sdh1 to /dev/md1 as 12
>> mdadm: added /dev/sdl1 to /dev/md1 as 0
>> mdadm: /dev/md1 assembled from 10 drives - not enough to start the array.
>
> That looks a lot more sensible.
> So that array expects 13 drives, but you only have 10.
> You'll need to find at least 1 more (preferably 3 more) to have any chance of
> success.
>
>>
>>
>> Should I try to force it?  Worried it might make things worse.
>
> force won't help until you find those other devices.
>
> 'force' is unlikely to make things worse.  It does the best it can.  The
> reason that you need to actually give "--force" (rather than mdadm always
> doing the best it case) is that you need to know that something has gone
> wrong and so not to trust the contents of the array until you have verified
> that your data is safe (most of it will be, but no promises).
>
> NeilBrown
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html