accidently pulled to many devices, raid6 wont start.

accidently pulled to many devices, raid6 wont start.

[Wilson Jonathan]

I have tried a --assemble force..

the result was it could not because the devices said it was ignored.

root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --assemble
-v /dev/md5 
mdadm: looking for devices for /dev/md5
mdadm: no RAID superblock on /dev/md/7
mdadm: no RAID superblock on /dev/md6
mdadm: no RAID superblock on /dev/md/4
mdadm: no RAID superblock on /dev/md/3
mdadm: no RAID superblock on /dev/md/2
mdadm: no RAID superblock on /dev/md/1
mdadm: no RAID superblock on /dev/md/0
mdadm: /dev/sdh7 has wrong uuid.
mdadm: /dev/sdh6 has wrong uuid.
mdadm: /dev/sdh5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdh4
mdadm: /dev/sdh3 has wrong uuid.
mdadm: /dev/sdh2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdh1
mdadm: no RAID superblock on /dev/sdh
mdadm: /dev/sdg7 has wrong uuid.
mdadm: /dev/sdg6 has wrong uuid.
mdadm: /dev/sdg5 has wrong uuid.
mdadm: no RAID superblock on /dev/sdg4
mdadm: /dev/sdg3 has wrong uuid.
mdadm: /dev/sdg2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdg1
mdadm: no RAID superblock on /dev/sdg
mdadm: /dev/sdf1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdf
mdadm: /dev/sde1 has wrong uuid.
mdadm: no RAID superblock on /dev/sde
mdadm: /dev/sdd1 has wrong uuid.
mdadm: no RAID superblock on /dev/sdd
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdb5 has wrong uuid.
mdadm: /dev/sdb4 has wrong uuid.
mdadm: /dev/sdb3 has wrong uuid.
mdadm: /dev/sdb2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdb1
mdadm: no RAID superblock on /dev/sdb
mdadm: /dev/sda1 has wrong uuid.
mdadm: no RAID superblock on /dev/sda
mdadm: /dev/sdf6 is identified as a member of /dev/md5, slot 2.
mdadm: /dev/sde6 is identified as a member of /dev/md5, slot 1.
mdadm: /dev/sdd6 is identified as a member of /dev/md5, slot 0.
mdadm: /dev/sdb6 is identified as a member of /dev/md5, slot 4.
mdadm: /dev/sda6 is identified as a member of /dev/md5, slot 3.
mdadm: ignoring /dev/sde6 as it reports /dev/sdf6 as failed
mdadm: ignoring /dev/sda6 as it reports /dev/sdf6 as failed
mdadm: ignoring /dev/sdb6 as it reports /dev/sdf6 as failed
mdadm: no uptodate device for slot 1 of /dev/md5
mdadm: added /dev/sdf6 to /dev/md5 as 2
mdadm: no uptodate device for slot 3 of /dev/md5
mdadm: no uptodate device for slot 4 of /dev/md5
mdadm: no uptodate device for slot 5 of /dev/md5
mdadm: added /dev/sdd6 to /dev/md5 as 0
mdadm: /dev/md5 assembled from 2 drives - not enough to start the array.
root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --stop /dev/md5


So i checked count:-

root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm
--examine /dev/sd[a-f]6 | egrep 'Event|/dev/sd'
/dev/sda6:
         Events : 260321
/dev/sdb6:
         Events : 260321
/dev/sdd6:
         Events : 260321
/dev/sde6:
         Events : 260321
/dev/sdf6:
         Events : 260321

My full scan of the partiotions was:-

/dev/sda6:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 9ff95b8b:ba34ad95:aa7cb806:169246f2
           Name : PartedMagic:6
  Creation Time : Fri Nov 25 16:30:19 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 1833569679 (874.31 GiB 938.79 GB)
     Array Size : 3667138560 (3497.26 GiB 3755.15 GB)
  Used Dev Size : 1833569280 (874.31 GiB 938.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : f773d872:8237b0f7:1a8fb43e:545ef37d

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Dec  4 23:07:34 2013
       Checksum : 75d8e87e - correct
         Events : 260321

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 3
   Array State : AA.AA. ('A' == active, '.' == missing)
/dev/sdb6:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 9ff95b8b:ba34ad95:aa7cb806:169246f2
           Name : PartedMagic:6
  Creation Time : Fri Nov 25 16:30:19 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 1833569287 (874.31 GiB 938.79 GB)
     Array Size : 3667138560 (3497.26 GiB 3755.15 GB)
  Used Dev Size : 1833569280 (874.31 GiB 938.79 GB)
    Data Offset : 1928 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : dbda8da3:0d5e0968:e732a312:88a04c2a

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Dec  4 23:15:16 2013
       Checksum : 97305bb9 - correct
         Events : 260321

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 4
   Array State : AA..A. ('A' == active, '.' == missing)
/dev/sdd6:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 9ff95b8b:ba34ad95:aa7cb806:169246f2
           Name : PartedMagic:6
  Creation Time : Fri Nov 25 16:30:19 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 1833569679 (874.31 GiB 938.79 GB)
     Array Size : 3667138560 (3497.26 GiB 3755.15 GB)
  Used Dev Size : 1833569280 (874.31 GiB 938.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : b451254c:dbc5f0fe:5e182208:6bd93c2c

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Dec  4 23:15:16 2013
       Checksum : ce1c5ac1 - correct
         Events : 260321

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AAAAA. ('A' == active, '.' == missing)
/dev/sde6:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 9ff95b8b:ba34ad95:aa7cb806:169246f2
           Name : PartedMagic:6
  Creation Time : Fri Nov 25 16:30:19 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 1833569679 (874.31 GiB 938.79 GB)
     Array Size : 3667138560 (3497.26 GiB 3755.15 GB)
  Used Dev Size : 1833569280 (874.31 GiB 938.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : c0200a09:fac761e5:43a54346:34597026

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Dec  4 23:15:16 2013
       Checksum : a9c9388b - correct
         Events : 260321

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : AA..A. ('A' == active, '.' == missing)
/dev/sdf6:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 9ff95b8b:ba34ad95:aa7cb806:169246f2
           Name : PartedMagic:6
  Creation Time : Fri Nov 25 16:30:19 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 1833569679 (874.31 GiB 938.79 GB)
     Array Size : 3667138560 (3497.26 GiB 3755.15 GB)
  Used Dev Size : 1833569280 (874.31 GiB 938.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 19b46e87:20182485:2f386b4e:b7f52203

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Dec  4 23:05:48 2013
       Checksum : acc8494b - correct
         Events : 260321

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 2
   Array State : AAAAA. ('A' == active, '.' == missing)

I tried to do a force create:-

root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --create
--assume-clean --level=6 --raid-devices=6 /dev/md5 /dev/sdd6 /dev/sde6
missing /dev/sdf6 /dev/sda6 /dev/sdb6
mdadm: /dev/sdd6 appears to be part of a raid array:
    level=raid6 devices=6 ctime=Fri Nov 25 16:30:19 2011
mdadm: /dev/sde6 appears to be part of a raid array:
    level=raid6 devices=6 ctime=Fri Nov 25 16:30:19 2011
mdadm: /dev/sdf6 appears to be part of a raid array:
    level=raid6 devices=6 ctime=Fri Nov 25 16:30:19 2011
mdadm: /dev/sda6 appears to be part of a raid array:
    level=raid6 devices=6 ctime=Fri Nov 25 16:30:19 2011
mdadm: /dev/sdb6 appears to be part of a raid array:
    level=raid6 devices=6 ctime=Fri Nov 25 16:30:19 2011
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md5 started.
root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --stop /dev/md5


but when I looked at it, it was wrongly partitioned, so stopped it...

So tried again, with a different (valid) chunk...


root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --create
--assume-clean --level=6 --raid-devices=6
--chunk=64 /dev/md5 /dev/sdd6 /dev/sde6
missing /dev/sdf6 /dev/sda6 /dev/sdb6
mdadm: /dev/sdd6 appears to be part of a raid array:
    level=raid6 devices=6 ctime=Wed Dec  4 23:59:23 2013
mdadm: /dev/sde6 appears to be part of a raid array:
    level=raid6 devices=6 ctime=Wed Dec  4 23:59:23 2013
mdadm: /dev/sdf6 appears to be part of a raid array:
    level=raid6 devices=6 ctime=Wed Dec  4 23:59:23 2013
mdadm: /dev/sda6 appears to be part of a raid array:
    level=raid6 devices=6 ctime=Wed Dec  4 23:59:23 2013
mdadm: /dev/sdb6 appears to be part of a raid array:
    level=raid6 devices=6 ctime=Wed Dec  4 23:59:23 2013
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md5 started.
root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --stop /dev/md5

but still the partition was all wrong... it should be a single
partition, but its showing multiple partitions... and is
reporting /dev/md5p3 "warning miss aligned"

I think I should, and have, stop doing anything else, till I have some
help :-/

Re: accidently pulled to many devices, raid6 wont start.

[Wilson Jonathan]

I should also state I can put the originall pulled drive back in if that
would be of any help, and re-run the -examine if that might be more
sensible as the drive should be ok.


On Thu, 2013-12-05 at 00:30 +0000, Wilson Jonathan wrote:
> I have tried a --assemble force..
> 
> the result was it could not because the devices said it was ignored.
> 
> root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --assemble
> -v /dev/md5 
> mdadm: looking for devices for /dev/md5
> mdadm: no RAID superblock on /dev/md/7
> mdadm: no RAID superblock on /dev/md6
> mdadm: no RAID superblock on /dev/md/4
> mdadm: no RAID superblock on /dev/md/3
> mdadm: no RAID superblock on /dev/md/2
> mdadm: no RAID superblock on /dev/md/1
> mdadm: no RAID superblock on /dev/md/0
> mdadm: /dev/sdh7 has wrong uuid.
> mdadm: /dev/sdh6 has wrong uuid.
> mdadm: /dev/sdh5 has wrong uuid.
> mdadm: no RAID superblock on /dev/sdh4
> mdadm: /dev/sdh3 has wrong uuid.
> mdadm: /dev/sdh2 has wrong uuid.
> mdadm: no RAID superblock on /dev/sdh1
> mdadm: no RAID superblock on /dev/sdh
> mdadm: /dev/sdg7 has wrong uuid.
> mdadm: /dev/sdg6 has wrong uuid.
> mdadm: /dev/sdg5 has wrong uuid.
> mdadm: no RAID superblock on /dev/sdg4
> mdadm: /dev/sdg3 has wrong uuid.
> mdadm: /dev/sdg2 has wrong uuid.
> mdadm: no RAID superblock on /dev/sdg1
> mdadm: no RAID superblock on /dev/sdg
> mdadm: /dev/sdf1 has wrong uuid.
> mdadm: no RAID superblock on /dev/sdf
> mdadm: /dev/sde1 has wrong uuid.
> mdadm: no RAID superblock on /dev/sde
> mdadm: /dev/sdd1 has wrong uuid.
> mdadm: no RAID superblock on /dev/sdd
> mdadm: no RAID superblock on /dev/sdc
> mdadm: /dev/sdb5 has wrong uuid.
> mdadm: /dev/sdb4 has wrong uuid.
> mdadm: /dev/sdb3 has wrong uuid.
> mdadm: /dev/sdb2 has wrong uuid.
> mdadm: no RAID superblock on /dev/sdb1
> mdadm: no RAID superblock on /dev/sdb
> mdadm: /dev/sda1 has wrong uuid.
> mdadm: no RAID superblock on /dev/sda
> mdadm: /dev/sdf6 is identified as a member of /dev/md5, slot 2.
> mdadm: /dev/sde6 is identified as a member of /dev/md5, slot 1.
> mdadm: /dev/sdd6 is identified as a member of /dev/md5, slot 0.
> mdadm: /dev/sdb6 is identified as a member of /dev/md5, slot 4.
> mdadm: /dev/sda6 is identified as a member of /dev/md5, slot 3.
> mdadm: ignoring /dev/sde6 as it reports /dev/sdf6 as failed
> mdadm: ignoring /dev/sda6 as it reports /dev/sdf6 as failed
> mdadm: ignoring /dev/sdb6 as it reports /dev/sdf6 as failed
> mdadm: no uptodate device for slot 1 of /dev/md5
> mdadm: added /dev/sdf6 to /dev/md5 as 2
> mdadm: no uptodate device for slot 3 of /dev/md5
> mdadm: no uptodate device for slot 4 of /dev/md5
> mdadm: no uptodate device for slot 5 of /dev/md5
> mdadm: added /dev/sdd6 to /dev/md5 as 0
> mdadm: /dev/md5 assembled from 2 drives - not enough to start the array.
> root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --stop /dev/md5
> 
> 
> So i checked count:-
> 
> root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm
> --examine /dev/sd[a-f]6 | egrep 'Event|/dev/sd'
> /dev/sda6:
>          Events : 260321
> /dev/sdb6:
>          Events : 260321
> /dev/sdd6:
>          Events : 260321
> /dev/sde6:
>          Events : 260321
> /dev/sdf6:
>          Events : 260321
> 
> My full scan of the partiotions was:-
> 
> /dev/sda6:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x1
>      Array UUID : 9ff95b8b:ba34ad95:aa7cb806:169246f2
>            Name : PartedMagic:6
>   Creation Time : Fri Nov 25 16:30:19 2011
>      Raid Level : raid6
>    Raid Devices : 6
> 
>  Avail Dev Size : 1833569679 (874.31 GiB 938.79 GB)
>      Array Size : 3667138560 (3497.26 GiB 3755.15 GB)
>   Used Dev Size : 1833569280 (874.31 GiB 938.79 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>           State : active
>     Device UUID : f773d872:8237b0f7:1a8fb43e:545ef37d
> 
> Internal Bitmap : 8 sectors from superblock
>     Update Time : Wed Dec  4 23:07:34 2013
>        Checksum : 75d8e87e - correct
>          Events : 260321
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>    Device Role : Active device 3
>    Array State : AA.AA. ('A' == active, '.' == missing)
> /dev/sdb6:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x1
>      Array UUID : 9ff95b8b:ba34ad95:aa7cb806:169246f2
>            Name : PartedMagic:6
>   Creation Time : Fri Nov 25 16:30:19 2011
>      Raid Level : raid6
>    Raid Devices : 6
> 
>  Avail Dev Size : 1833569287 (874.31 GiB 938.79 GB)
>      Array Size : 3667138560 (3497.26 GiB 3755.15 GB)
>   Used Dev Size : 1833569280 (874.31 GiB 938.79 GB)
>     Data Offset : 1928 sectors
>    Super Offset : 8 sectors
>           State : clean
>     Device UUID : dbda8da3:0d5e0968:e732a312:88a04c2a
> 
> Internal Bitmap : 8 sectors from superblock
>     Update Time : Wed Dec  4 23:15:16 2013
>        Checksum : 97305bb9 - correct
>          Events : 260321
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>    Device Role : Active device 4
>    Array State : AA..A. ('A' == active, '.' == missing)
> /dev/sdd6:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x1
>      Array UUID : 9ff95b8b:ba34ad95:aa7cb806:169246f2
>            Name : PartedMagic:6
>   Creation Time : Fri Nov 25 16:30:19 2011
>      Raid Level : raid6
>    Raid Devices : 6
> 
>  Avail Dev Size : 1833569679 (874.31 GiB 938.79 GB)
>      Array Size : 3667138560 (3497.26 GiB 3755.15 GB)
>   Used Dev Size : 1833569280 (874.31 GiB 938.79 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>           State : clean
>     Device UUID : b451254c:dbc5f0fe:5e182208:6bd93c2c
> 
> Internal Bitmap : 8 sectors from superblock
>     Update Time : Wed Dec  4 23:15:16 2013
>        Checksum : ce1c5ac1 - correct
>          Events : 260321
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>    Device Role : Active device 0
>    Array State : AAAAA. ('A' == active, '.' == missing)
> /dev/sde6:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x1
>      Array UUID : 9ff95b8b:ba34ad95:aa7cb806:169246f2
>            Name : PartedMagic:6
>   Creation Time : Fri Nov 25 16:30:19 2011
>      Raid Level : raid6
>    Raid Devices : 6
> 
>  Avail Dev Size : 1833569679 (874.31 GiB 938.79 GB)
>      Array Size : 3667138560 (3497.26 GiB 3755.15 GB)
>   Used Dev Size : 1833569280 (874.31 GiB 938.79 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>           State : clean
>     Device UUID : c0200a09:fac761e5:43a54346:34597026
> 
> Internal Bitmap : 8 sectors from superblock
>     Update Time : Wed Dec  4 23:15:16 2013
>        Checksum : a9c9388b - correct
>          Events : 260321
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>    Device Role : Active device 1
>    Array State : AA..A. ('A' == active, '.' == missing)
> /dev/sdf6:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x1
>      Array UUID : 9ff95b8b:ba34ad95:aa7cb806:169246f2
>            Name : PartedMagic:6
>   Creation Time : Fri Nov 25 16:30:19 2011
>      Raid Level : raid6
>    Raid Devices : 6
> 
>  Avail Dev Size : 1833569679 (874.31 GiB 938.79 GB)
>      Array Size : 3667138560 (3497.26 GiB 3755.15 GB)
>   Used Dev Size : 1833569280 (874.31 GiB 938.79 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>           State : active
>     Device UUID : 19b46e87:20182485:2f386b4e:b7f52203
> 
> Internal Bitmap : 8 sectors from superblock
>     Update Time : Wed Dec  4 23:05:48 2013
>        Checksum : acc8494b - correct
>          Events : 260321
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>    Device Role : Active device 2
>    Array State : AAAAA. ('A' == active, '.' == missing)
> 
> I tried to do a force create:-
> 
> root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --create
> --assume-clean --level=6 --raid-devices=6 /dev/md5 /dev/sdd6 /dev/sde6
> missing /dev/sdf6 /dev/sda6 /dev/sdb6
> mdadm: /dev/sdd6 appears to be part of a raid array:
>     level=raid6 devices=6 ctime=Fri Nov 25 16:30:19 2011
> mdadm: /dev/sde6 appears to be part of a raid array:
>     level=raid6 devices=6 ctime=Fri Nov 25 16:30:19 2011
> mdadm: /dev/sdf6 appears to be part of a raid array:
>     level=raid6 devices=6 ctime=Fri Nov 25 16:30:19 2011
> mdadm: /dev/sda6 appears to be part of a raid array:
>     level=raid6 devices=6 ctime=Fri Nov 25 16:30:19 2011
> mdadm: /dev/sdb6 appears to be part of a raid array:
>     level=raid6 devices=6 ctime=Fri Nov 25 16:30:19 2011
> Continue creating array? y
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md5 started.
> root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --stop /dev/md5
> 
> 
> but when I looked at it, it was wrongly partitioned, so stopped it...
> 
> So tried again, with a different (valid) chunk...
> 
> 
> root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --create
> --assume-clean --level=6 --raid-devices=6
> --chunk=64 /dev/md5 /dev/sdd6 /dev/sde6
> missing /dev/sdf6 /dev/sda6 /dev/sdb6
> mdadm: /dev/sdd6 appears to be part of a raid array:
>     level=raid6 devices=6 ctime=Wed Dec  4 23:59:23 2013
> mdadm: /dev/sde6 appears to be part of a raid array:
>     level=raid6 devices=6 ctime=Wed Dec  4 23:59:23 2013
> mdadm: /dev/sdf6 appears to be part of a raid array:
>     level=raid6 devices=6 ctime=Wed Dec  4 23:59:23 2013
> mdadm: /dev/sda6 appears to be part of a raid array:
>     level=raid6 devices=6 ctime=Wed Dec  4 23:59:23 2013
> mdadm: /dev/sdb6 appears to be part of a raid array:
>     level=raid6 devices=6 ctime=Wed Dec  4 23:59:23 2013
> Continue creating array? y
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md5 started.
> root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --stop /dev/md5
> 
> but still the partition was all wrong... it should be a single
> partition, but its showing multiple partitions... and is
> reporting /dev/md5p3 "warning miss aligned"
> 
> I think I should, and have, stop doing anything else, till I have some
> help :-/
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: accidently pulled to many devices, raid6 wont start.

[Bernd Schubert]

On 12/05/2013 01:30 AM, Wilson Jonathan wrote:
> mdadm: /dev/sdf6 is identified as a member of /dev/md5, slot 2.
> mdadm: /dev/sde6 is identified as a member of /dev/md5, slot 1.
> mdadm: /dev/sdd6 is identified as a member of /dev/md5, slot 0.
> mdadm: /dev/sdb6 is identified as a member of /dev/md5, slot 4.
> mdadm: /dev/sda6 is identified as a member of /dev/md5, slot 3.
> mdadm: ignoring /dev/sde6 as it reports /dev/sdf6 as failed

[...]

> So tried again, with a different (valid) chunk...
>
>
> root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --create
> --assume-clean --level=6 --raid-devices=6
> --chunk=64 /dev/md5 /dev/sdd6 /dev/sde6
> missing /dev/sdf6 /dev/sda6 /dev/sdb6

Why are you using this order? Missing seems to be at the wrong place?


Cheers,
Bernd

Re: accidently pulled to many devices, raid6 wont start.

[Wilson Jonathan]

On Thu, 2013-12-05 at 14:36 +0100, Bernd Schubert wrote:
> On 12/05/2013 01:30 AM, Wilson Jonathan wrote:
> > mdadm: /dev/sdf6 is identified as a member of /dev/md5, slot 2.
> > mdadm: /dev/sde6 is identified as a member of /dev/md5, slot 1.
> > mdadm: /dev/sdd6 is identified as a member of /dev/md5, slot 0.
> > mdadm: /dev/sdb6 is identified as a member of /dev/md5, slot 4.
> > mdadm: /dev/sda6 is identified as a member of /dev/md5, slot 3.
> > mdadm: ignoring /dev/sde6 as it reports /dev/sdf6 as failed
> 
> [...]
> 
> > So tried again, with a different (valid) chunk...
> >
> >
> > root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --create
> > --assume-clean --level=6 --raid-devices=6
> > --chunk=64 /dev/md5 /dev/sdd6 /dev/sde6
> > missing /dev/sdf6 /dev/sda6 /dev/sdb6
> 
> Why are you using this order? Missing seems to be at the wrong place?
> 
> 
> Cheers,
> Bernd

Arr yes, I seem to have got the order wrong... before I continue and do
further damage; I put the original pulled drive (sdc) back in and re-ran
examine. Obviously I have corrupted the raid layout data on the other
disks (a,b,d,e,f) but here is the sdc examine output.

/dev/sdc6:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 9ff95b8b:ba34ad95:aa7cb806:169246f2
           Name : PartedMagic:6
  Creation Time : Fri Nov 25 16:30:19 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 1833569679 (874.31 GiB 938.79 GB)
     Array Size : 3667138560 (3497.26 GiB 3755.15 GB)
  Used Dev Size : 1833569280 (874.31 GiB 938.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 4ce632fb:d506da60:120cac8e:4433efc4

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Dec  4 23:04:40 2013
       Checksum : fe4e7b1f - correct
         Events : 260243

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 5
   Array State : AAAAAA ('A' == active, '.' == missing)


> 
> 

I mistook the order (a-b-c-d-) with the position number in the array.

If I'm reading this correctly, Active device 5 means it should be at
position 5, or the sixth disk in the array.

I also note it has a bitmap on it, something the other drives now lack
as it was not stated in the original forced create, so not sure how that
will affect things and the event count is lower, 260243, than the other
devices, 260321, before the array became corrupted (that said I know
exactly which files were being updated on the file system, so if I can
get it back I will just delete them and re-create).

Hopefully I have not totally killed the array by my mistakes.

Re: accidently pulled to many devices, raid6 wont start.

[Bernd Schubert]

On 12/05/2013 03:20 PM, Wilson Jonathan wrote:
> On Thu, 2013-12-05 at 14:36 +0100, Bernd Schubert wrote:
>> On 12/05/2013 01:30 AM, Wilson Jonathan wrote:
>>> mdadm: /dev/sdf6 is identified as a member of /dev/md5, slot 2.
>>> mdadm: /dev/sde6 is identified as a member of /dev/md5, slot 1.
>>> mdadm: /dev/sdd6 is identified as a member of /dev/md5, slot 0.
>>> mdadm: /dev/sdb6 is identified as a member of /dev/md5, slot 4.
>>> mdadm: /dev/sda6 is identified as a member of /dev/md5, slot 3.
>>> mdadm: ignoring /dev/sde6 as it reports /dev/sdf6 as failed
>>
>> [...]
>>
>>> So tried again, with a different (valid) chunk...
>>>
>>>
>>> root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --create
>>> --assume-clean --level=6 --raid-devices=6
>>> --chunk=64 /dev/md5 /dev/sdd6 /dev/sde6
>>> missing /dev/sdf6 /dev/sda6 /dev/sdb6
>>
>> Why are you using this order? Missing seems to be at the wrong place?
>>
>>
>> Cheers,
>> Bernd
>
> Arr yes, I seem to have got the order wrong... before I continue and do
> further damage; I put the original pulled drive (sdc) back in and re-ran
> examine. Obviously I have corrupted the raid layout data on the other
> disks (a,b,d,e,f) but here is the sdc examine output.
>
> /dev/sdc6:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x1
>       Array UUID : 9ff95b8b:ba34ad95:aa7cb806:169246f2
>             Name : PartedMagic:6
>    Creation Time : Fri Nov 25 16:30:19 2011
>       Raid Level : raid6
>     Raid Devices : 6
>
>   Avail Dev Size : 1833569679 (874.31 GiB 938.79 GB)
>       Array Size : 3667138560 (3497.26 GiB 3755.15 GB)
>    Used Dev Size : 1833569280 (874.31 GiB 938.79 GB)
>      Data Offset : 2048 sectors
>     Super Offset : 8 sectors
>            State : active
>      Device UUID : 4ce632fb:d506da60:120cac8e:4433efc4
>
> Internal Bitmap : 8 sectors from superblock
>      Update Time : Wed Dec  4 23:04:40 2013
>         Checksum : fe4e7b1f - correct
>           Events : 260243
>
>           Layout : left-symmetric
>       Chunk Size : 64K
>
>     Device Role : Active device 5
>     Array State : AAAAAA ('A' == active, '.' == missing)
>
>
>>
>>
>
> I mistook the order (a-b-c-d-) with the position number in the array.
>
> If I'm reading this correctly, Active device 5 means it should be at
> position 5, or the sixth disk in the array.

Yes, and the intial kernel output also suggest that.

>
> I also note it has a bitmap on it, something the other drives now lack
> as it was not stated in the original forced create, so not sure how that
> will affect things and the event count is lower, 260243, than the other
> devices, 260321, before the array became corrupted (that said I know
> exactly which files were being updated on the file system, so if I can
> get it back I will just delete them and re-create).
>
> Hopefully I have not totally killed the array by my mistakes.

I would simply try again with the correct order - 'missing' as last 
argument. Then check your data, i.e. test if it mounts (read-only) and 
run fsck (read-only).

I'm also a bit suprised why you went the '--create' way at all, why 
didn't you simply try to assemble with 4 drives only?


Cheers,
Bernd

Re: accidently pulled to many devices, raid6 wont start.

[Wilson Jonathan]

> > Hopefully I have not totally killed the array by my mistakes.
> 
> I would simply try again with the correct order - 'missing' as last 
> argument. Then check your data, i.e. test if it mounts (read-only) and 
> run fsck (read-only).
> 
> I'm also a bit suprised why you went the '--create' way at all, why 
> didn't you simply try to assemble with 4 drives only?

A combination of factors, most notably this being a home server for
myself and my son, a few glasses of wine, and cries of "dad I can't get
to my games folder" from his bedroom... then reading misleading web
pages suggesting that if assemble fails to use create and compounding
the issue by not understanding that an assemble can be used with force
and a list of devices (at least as far as I understand) which no doubt I
would have still put in the wrong order in my haste to get it back up
and running...

With hindsight I should have just turned the thing off, gone to bed, and
posted here in the morning before trying anything.

I shall try as you suggest, run the create with the disks in the correct
order and ignore re-inserted sdc and see what happens once I have got
back from the shops.

Thanks for the help so far.

> 
> 
> Cheers,
> Bernd
> 

Re: accidently pulled to many devices, raid6 wont start.

[Wilson Jonathan]

Still the same problem, lists non partitioned space, unknown, unknown,
un partitioned.

I wonder if the problem is because i'm not re-creating it with bitmap
file, which is listed as starting 8 sectors from superblock, so with it
not being there the partition table is being read from this space and
not further along the space.

Infact, looking at the re-created superblock there is a change...

/dev/sde6:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 1c413162:25c7b4af:7b305603:1ce49fed
           Name : BorgCUBE:5  (local to host BorgCUBE)
  Creation Time : Thu Dec  5 00:28:11 2013
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 1833309583 (874.19 GiB 938.65 GB)
     Array Size : 3666617344 (3496.76 GiB 3754.62 GB)
  Used Dev Size : 1833308672 (874.19 GiB 938.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : de841586:f82b22ed:1a234ae7:57fbb36a

    Update Time : Thu Dec  5 00:28:11 2013
       Checksum : 9dcbeec1 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 64K

The "data offset" is 262144 where as the sdc (original) is 2048

/dev/sdc6:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 9ff95b8b:ba34ad95:aa7cb806:169246f2
           Name : PartedMagic:6
  Creation Time : Fri Nov 25 16:30:19 2011
     Raid Level : raid6
   Raid Devices : 6

 Avail Dev Size : 1833569679 (874.31 GiB 938.79 GB)
     Array Size : 3667138560 (3497.26 GiB 3755.15 GB)
  Used Dev Size : 1833569280 (874.31 GiB 938.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 4ce632fb:d506da60:120cac8e:4433efc4

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Dec  4 23:04:40 2013
       Checksum : fe4e7b1f - correct
         Events : 260243

         Layout : left-symmetric
     Chunk Size : 64K

Re: accidently pulled to many devices, raid6 wont start.

[Bernd Schubert]

On 12/05/2013 04:48 PM, Wilson Jonathan wrote:
> Still the same problem, lists non partitioned space, unknown, unknown,
> un partitioned.
>
> I wonder if the problem is because i'm not re-creating it with bitmap
> file, which is listed as starting 8 sectors from superblock, so with it
> not being there the partition table is being read from this space and
> not further along the space.

The bitmap is supposed to be in the gap between data and superblock.

>
> Infact, looking at the re-created superblock there is a change...
>
> /dev/sde6:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x0
>       Array UUID : 1c413162:25c7b4af:7b305603:1ce49fed
>             Name : BorgCUBE:5  (local to host BorgCUBE)
>    Creation Time : Thu Dec  5 00:28:11 2013
>       Raid Level : raid6
>     Raid Devices : 6
>
>   Avail Dev Size : 1833309583 (874.19 GiB 938.65 GB)
>       Array Size : 3666617344 (3496.76 GiB 3754.62 GB)
>    Used Dev Size : 1833308672 (874.19 GiB 938.65 GB)
>      Data Offset : 262144 sectors
>     Super Offset : 8 sectors
>            State : clean
>      Device UUID : de841586:f82b22ed:1a234ae7:57fbb36a
>
>      Update Time : Thu Dec  5 00:28:11 2013
>         Checksum : 9dcbeec1 - correct
>           Events : 0
>
>           Layout : left-symmetric
>       Chunk Size : 64K
>
> The "data offset" is 262144 where as the sdc (original) is 2048

Seems to be a new mdadm version with different default values. Another 
reason not to use '--create' unless there is really no other way. Try to 
add --data-offset=1M.


Cheers,
Bernd


PS: Btw, for assemble the order of devices doesn't matter, the kernel 
know the correct order from their superblocks...

Re: accidently pulled to many devices, raid6 wont start.

[Wilson Jonathan]

On Thu, 2013-12-05 at 17:00 +0100, Bernd Schubert wrote:
> On 12/05/2013 04:48 PM, Wilson Jonathan wrote:
> > Still the same problem, lists non partitioned space, unknown, unknown,
> > un partitioned.
> >
> > I wonder if the problem is because i'm not re-creating it with bitmap
> > file, which is listed as starting 8 sectors from superblock, so with it
> > not being there the partition table is being read from this space and
> > not further along the space.
> 
> The bitmap is supposed to be in the gap between data and superblock.
> 
> >
> > Infact, looking at the re-created superblock there is a change...
> >
> > /dev/sde6:
> >            Magic : a92b4efc
> >          Version : 1.2
> >      Feature Map : 0x0
> >       Array UUID : 1c413162:25c7b4af:7b305603:1ce49fed
> >             Name : BorgCUBE:5  (local to host BorgCUBE)
> >    Creation Time : Thu Dec  5 00:28:11 2013
> >       Raid Level : raid6
> >     Raid Devices : 6
> >
> >   Avail Dev Size : 1833309583 (874.19 GiB 938.65 GB)
> >       Array Size : 3666617344 (3496.76 GiB 3754.62 GB)
> >    Used Dev Size : 1833308672 (874.19 GiB 938.65 GB)
> >      Data Offset : 262144 sectors
> >     Super Offset : 8 sectors
> >            State : clean
> >      Device UUID : de841586:f82b22ed:1a234ae7:57fbb36a
> >
> >      Update Time : Thu Dec  5 00:28:11 2013
> >         Checksum : 9dcbeec1 - correct
> >           Events : 0
> >
> >           Layout : left-symmetric
> >       Chunk Size : 64K
> >
> > The "data offset" is 262144 where as the sdc (original) is 2048
> 
> Seems to be a new mdadm version with different default values. Another 
> reason not to use '--create' unless there is really no other way. Try to 
> add --data-offset=1M.

My version of mdadm does not have --data-offset...

root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --create
--assume-clean --level=6 --raid-devices=6 --chunk=64
--data-offset=1M /dev/md5 /dev/sdd6 /dev/sde6 /dev/sdf6 /dev/sda6
missing missing 
mdadm: unrecognized option '--data-offset=1M'

I also double checked the original, pre-corrupted, examine and found
that /dev/sdb6 had 1928 sectors (was a new 3tb drive, using a partition
on it, which had a slightly larger size, to replace the original
partition on a 1tb disk)

sda6:3, sdd6:0, sde6:1, sdf6:2 are all at 2048 sectors, so if I can
somehow create md5 using just these 4 disks it should be ok even if it
has no redundancy, assuming it works and I can get hold of a mdadm with
offset (I have git, and have built from source before if it comes to
that... but a binary would be more helpful, or perhaps a mdadm that had
the original 2048 offset?) then clear the superblock from b and c and
re-add them to re-build the redundancy.

Thanks.

> 
> 
> Cheers,
> Bernd
> 
> 
> PS: Btw, for assemble the order of devices doesn't matter, the kernel 
> know the correct order from their superblocks...
> 

Re: accidently pulled to many devices, raid6 wont start.

[Bernd Schubert]

On 12/05/2013 05:36 PM, Wilson Jonathan wrote:
> On Thu, 2013-12-05 at 17:00 +0100, Bernd Schubert wrote:
>> On 12/05/2013 04:48 PM, Wilson Jonathan wrote:
>>> Still the same problem, lists non partitioned space, unknown, unknown,
>>> un partitioned.
>>>
>>> I wonder if the problem is because i'm not re-creating it with bitmap
>>> file, which is listed as starting 8 sectors from superblock, so with it
>>> not being there the partition table is being read from this space and
>>> not further along the space.
>>
>> The bitmap is supposed to be in the gap between data and superblock.
>>
>>>
>>> Infact, looking at the re-created superblock there is a change...
>>>
>>> /dev/sde6:
>>>             Magic : a92b4efc
>>>           Version : 1.2
>>>       Feature Map : 0x0
>>>        Array UUID : 1c413162:25c7b4af:7b305603:1ce49fed
>>>              Name : BorgCUBE:5  (local to host BorgCUBE)
>>>     Creation Time : Thu Dec  5 00:28:11 2013
>>>        Raid Level : raid6
>>>      Raid Devices : 6
>>>
>>>    Avail Dev Size : 1833309583 (874.19 GiB 938.65 GB)
>>>        Array Size : 3666617344 (3496.76 GiB 3754.62 GB)
>>>     Used Dev Size : 1833308672 (874.19 GiB 938.65 GB)
>>>       Data Offset : 262144 sectors
>>>      Super Offset : 8 sectors
>>>             State : clean
>>>       Device UUID : de841586:f82b22ed:1a234ae7:57fbb36a
>>>
>>>       Update Time : Thu Dec  5 00:28:11 2013
>>>          Checksum : 9dcbeec1 - correct
>>>            Events : 0
>>>
>>>            Layout : left-symmetric
>>>        Chunk Size : 64K
>>>
>>> The "data offset" is 262144 where as the sdc (original) is 2048
>>
>> Seems to be a new mdadm version with different default values. Another
>> reason not to use '--create' unless there is really no other way. Try to
>> add --data-offset=1M.
>
> My version of mdadm does not have --data-offset...
>
> root@BorgCUBE:/mnt/datastore/wilsonjonathan# mdadm --create
> --assume-clean --level=6 --raid-devices=6 --chunk=64
> --data-offset=1M /dev/md5 /dev/sdd6 /dev/sde6 /dev/sdf6 /dev/sda6
> missing missing
> mdadm: unrecognized option '--data-offset=1M'
>
> I also double checked the original, pre-corrupted, examine and found
> that /dev/sdb6 had 1928 sectors (was a new 3tb drive, using a partition
> on it, which had a slightly larger size, to replace the original
> partition on a 1tb disk)
>
> sda6:3, sdd6:0, sde6:1, sdf6:2 are all at 2048 sectors, so if I can
> somehow create md5 using just these 4 disks it should be ok even if it
> has no redundancy, assuming it works and I can get hold of a mdadm with

Sorry, I don't understand what you mean. sdb6 starts at 1928 sectors? 
Assuming you have either 4k or 512B hw-sector disks that shouldn't make 
a difference for alignment. If at all, I would correct it only once I 
would have full redundancy.

> offset (I have git, and have built from source before if it comes to
> that... but a binary would be more helpful, or perhaps a mdadm that had
> the original 2048 offset?) then clear the superblock from b and c and
> re-add them to re-build the redundancy.


Well, I could probably build mdadm for you (if I should have the chroot 
matching your distribution), but can't you just download an older or a 
recent version for your distribution?


Cheers,
Bernd

Re: accidently pulled to many devices, raid6 wont start.

[Wilson Jonathan]

> 
> Sorry, I don't understand what you mean. sdb6 starts at 1928 sectors? 
> Assuming you have either 4k or 512B hw-sector disks that shouldn't make 
> a difference for alignment. If at all, I would correct it only once I 
> would have full redundancy.

I think I've managed to get it back using the 4 disks I was happy
with...

> 
> > offset (I have git, and have built from source before if it comes to
> > that... but a binary would be more helpful, or perhaps a mdadm that had
> > the original 2048 offset?) then clear the superblock from b and c and
> > re-add them to re-build the redundancy.
> 
> 
> Well, I could probably build mdadm for you (if I should have the chroot 
> matching your distribution), but can't you just download an older or a 
> recent version for your distribution?

What I did was:-

wilsonjonathan@BorgCUBE:~$ git clone git://neil.brown.name/mdadm
Cloning into 'mdadm'...
remote: Counting objects: 12848, done.
remote: Compressing objects: 100% (4434/4434), done.
remote: Total 12848 (delta 9627), reused 11123 (delta 8405)
Receiving objects: 100% (12848/12848), 3.37 MiB | 184 KiB/s, done.
Resolving deltas: 100% (9627/9627), done.
wilsonjonathan@BorgCUBE:~$ cd mdadm
wilsonjonathan@BorgCUBE:~/mdadm$ make

<snip compliation stuff>

wilsonjonathan@BorgCUBE:~/mdadm$ ./mdadm --version
mdadm - v3.3-38-gb11fe74 - 03th December 2013

wilsonjonathan@BorgCUBE:~/mdadm$ sudo ./mdadm --create --assume-clean
--level=6 --raid-devices=6 --chunk=64
--data-offset=1M /dev/md5 /dev/sdd6 /dev/sde6 /dev/sdf6 /dev/sda6
missing missing
[sudo] password for wilsonjonathan: 
mdadm: /dev/sdd6 appears to be part of a raid array:
       level=raid6 devices=6 ctime=Thu Dec  5 15:33:34 2013
mdadm: /dev/sde6 appears to be part of a raid array:
       level=raid6 devices=6 ctime=Thu Dec  5 15:33:34 2013
mdadm: /dev/sdf6 appears to be part of a raid array:
       level=raid6 devices=6 ctime=Thu Dec  5 15:33:34 2013
mdadm: /dev/sda6 appears to be part of a raid array:
       level=raid6 devices=6 ctime=Thu Dec  5 15:33:34 2013
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md5 started.


Using disk manager in debian I can see md5, and the volume "md5mnt 3.8TB
ext4" as a single large partition.

I will run a fsck on it, then if thats ok will mount it read only and
have a poke around, then if thats ok will unmount it and add in the
missing drives/partitions.

Hopefully this means its all now ok *sigh of relief*

Thank you so much for your help so far on this, its been much
appreciated.

Jon.


> 
> 
> Cheers,
> Bernd
> 
>