Re: Question on how to mark drives on a particular raid as being good after a "failure"

[Richard Scobie <richard@sauce.co.nz>]

Using the --force option to assemble should do it.

You may need to stop the array in it's current state first, I'm not sure.

Regards,

Richard

Joe Landman wrote:
> Hi folks:
>
>    We've seen a "failure" ... whereby something like 10 drives
> simultaneously dropped out of a RAID6 array.  The drives haven't
> failed, we can see them, see the metadata on them, see the RAID
> partitions ... what we can't do is reassemble it into a RAID, as some
> of the drives are note listed as "Active"
>
>
> root@dv4:~# mdadm -v --assemble /dev/md0
> mdadm: looking for devices for /dev/md0
>
> ...
>
> mdadm: /dev/sdr3 is identified as a member of /dev/md0, slot 14.
> mdadm: /dev/sdy3 is identified as a member of /dev/md0, slot 15.
> mdadm: /dev/sdx3 is identified as a member of /dev/md0, slot -1.
> mdadm: /dev/sdw3 is identified as a member of /dev/md0, slot 21.
> mdadm: /dev/sdv3 is identified as a member of /dev/md0, slot 20.
> mdadm: /dev/sdu3 is identified as a member of /dev/md0, slot 19.
> mdadm: /dev/sdt3 is identified as a member of /dev/md0, slot 18.
> mdadm: /dev/sds3 is identified as a member of /dev/md0, slot 17.
> mdadm: /dev/sdq3 is identified as a member of /dev/md0, slot 7.
> mdadm: /dev/sdp3 is identified as a member of /dev/md0, slot 6.
> mdadm: /dev/sdo3 is identified as a member of /dev/md0, slot 5.
> mdadm: /dev/sdn3 is identified as a member of /dev/md0, slot 4.
> mdadm: /dev/sdm3 is identified as a member of /dev/md0, slot 3.
> mdadm: /dev/sdl3 is identified as a member of /dev/md0, slot 2.
> mdadm: /dev/sdk3 is identified as a member of /dev/md0, slot 1.
> mdadm: /dev/sdj3 is identified as a member of /dev/md0, slot 0.
> mdadm: /dev/sdc3 is identified as a member of /dev/md0, slot 9.
> mdadm: /dev/sdi3 is identified as a member of /dev/md0, slot 16.
> mdadm: /dev/sdh3 is identified as a member of /dev/md0, slot 13.
> mdadm: /dev/sdg3 is identified as a member of /dev/md0, slot -1.
> mdadm: /dev/sdf3 is identified as a member of /dev/md0, slot 12.
> mdadm: /dev/sde3 is identified as a member of /dev/md0, slot 11.
> mdadm: /dev/sdd3 is identified as a member of /dev/md0, slot 10.
> mdadm: /dev/sdb3 is identified as a member of /dev/md0, slot 8.
> mdadm: added /dev/sdj3 to /dev/md0 as 0
> mdadm: added /dev/sdk3 to /dev/md0 as 1
> mdadm: added /dev/sdl3 to /dev/md0 as 2
> mdadm: added /dev/sdm3 to /dev/md0 as 3
> mdadm: added /dev/sdn3 to /dev/md0 as 4
> mdadm: added /dev/sdo3 to /dev/md0 as 5
> mdadm: added /dev/sdp3 to /dev/md0 as 6
> mdadm: added /dev/sdq3 to /dev/md0 as 7
> mdadm: added /dev/sdc3 to /dev/md0 as 9
> mdadm: added /dev/sdd3 to /dev/md0 as 10
> mdadm: added /dev/sde3 to /dev/md0 as 11
> mdadm: added /dev/sdf3 to /dev/md0 as 12
> mdadm: added /dev/sdh3 to /dev/md0 as 13
> mdadm: added /dev/sdr3 to /dev/md0 as 14
> mdadm: added /dev/sdy3 to /dev/md0 as 15
> mdadm: added /dev/sdi3 to /dev/md0 as 16
> mdadm: added /dev/sds3 to /dev/md0 as 17
> mdadm: added /dev/sdt3 to /dev/md0 as 18
> mdadm: added /dev/sdu3 to /dev/md0 as 19
> mdadm: added /dev/sdv3 to /dev/md0 as 20
> mdadm: added /dev/sdw3 to /dev/md0 as 21
> mdadm: added /dev/sdx3 to /dev/md0 as -1
> mdadm: added /dev/sdg3 to /dev/md0 as -1
> mdadm: added /dev/sdb3 to /dev/md0 as 8
> mdadm: /dev/md0 assembled from 14 drives and 2 spares - not enough to
> start the array.
>
>
> root@dv4:~# cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
> [raid4] [raid10]
> md0 : inactive sdb3[8](S) sdg3[29](S) sdx3[27](S) sdw3[24](S)
> sdv3[20](S) sdu3[19](S) sdt3[18](S) sds3[25](S) sdi3[26](S)
> sdy3[23](S) sdr3[22](S) sdh3[28](S) sdf3[12](S) sde3[11](S)
> sdd3[10](S) sdc3[9](S) sdq3[7](S) sdp3[6](S) sdo3[5](S) sdn3[4](S)
> sdm3[3](S) sdl3[2](S) sdk3[1](S) sdj3[0](S)
>        34952381856 blocks super 1.2
>
> unused devices:<none>
>
> The array state is what is interesting.
>
> root@dv4:~# mdadm --examine /dev/sdj3
> /dev/sdj3:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x1
>       Array UUID : 94d790a9:619a5141:d98f1f0f:2511c5f9
>             Name : dv4:0  (local to host dv4)
>    Creation Time : Fri Jul 31 19:48:25 2009
>       Raid Level : raid6
>     Raid Devices : 22
>
>   Avail Dev Size : 2912698488 (1388.88 GiB 1491.30 GB)
>       Array Size : 58253967360 (27777.66 GiB 29826.03 GB)
>    Used Dev Size : 2912698368 (1388.88 GiB 1491.30 GB)
>      Data Offset : 272 sectors
>     Super Offset : 8 sectors
>            State : clean
>      Device UUID : 9af11157:115d8425:5f3cecf4:ea46725b
>
> Internal Bitmap : 8 sectors from superblock
>      Update Time : Sun Apr 18 23:29:12 2010
>         Checksum : 95ed75e2 - correct
>           Events : 752490
>
>           Layout : left-symmetric
>       Chunk Size : 1024K
>
>     Device Role : Active device 0
>     Array State : AAAAAAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing)
>
>
>
> root@dv4:~# mdadm --examine /dev/sdb3
> /dev/sdb3:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x1
>       Array UUID : 94d790a9:619a5141:d98f1f0f:2511c5f9
>             Name : dv4:0  (local to host dv4)
>    Creation Time : Fri Jul 31 19:48:25 2009
>       Raid Level : raid6
>     Raid Devices : 22
>
>   Avail Dev Size : 2912698488 (1388.88 GiB 1491.30 GB)
>       Array Size : 58253967360 (27777.66 GiB 29826.03 GB)
>    Used Dev Size : 2912698368 (1388.88 GiB 1491.30 GB)
>      Data Offset : 272 sectors
>     Super Offset : 8 sectors
>            State : active
>      Device UUID : e506e4a1:ac26540b:49aca00d:13077a06
>
> Internal Bitmap : 8 sectors from superblock
>      Update Time : Mon Apr 19 10:08:46 2010
>         Checksum : 8a3860bf - correct
>           Events : 770417
>
>           Layout : left-symmetric
>       Chunk Size : 1024K
>
>     Device Role : Active device 8
>     Array State : ........AAAAAAAAAAAAAA ('A' == active, '.' == missing)
>
>
> That is, the array state does not agree between two different groups
> of drives ... the 14 with the wrong array state, and the 10 with the
> correct array state.  Is there some magical incantation or mdadm
> command that will force the array to either "ignore" the array state
> metadata, and physically look at the drives for assembly, or clear the
> state and set it all as "good"?
>
> Thanks
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html