From mboxrd@z Thu Jan 1 00:00:00 1970 From: Neil Brown Subject: Re: md's fail to assemble correctly consistently at system startup - mdadm 3.1.2 and Ubuntu 10.04 Date: Mon, 9 Aug 2010 21:00:19 +1000 Message-ID: <20100809210019.3cc064ab@notabene> References: <20100808185806.69085415@notabene> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: In-Reply-To: Sender: linux-raid-owner@vger.kernel.org To: "fibreraid@gmail.com" Cc: linux-raid List-Id: linux-raid.ids On Sun, 8 Aug 2010 07:26:59 -0700 "fibreraid@gmail.com" wrote: > Thank you Neil for the reply and heads-up on 3.1.3. I will test that > immediately and report back my findings. >=20 > One potential issue I noticed is that Ubuntu Lucid's default kernel > configuration has CONFIG_MD_AUTODETECT enabled. I thought this featur= e > might conflict with udev, so I've attempted to disable this by adding > a parameter to my grub2 bootup: raid=3Dnoautodetect. But I am not sur= e > if this is effective. Do you think this kernel setting could also be = a > problem source? If you don't use "Linux raid autodetect" partition types (which you say= below that you don't) this CONFIG setting will have no effect at all. >=20 > Another method I was contemplating to avoid a potential locking issue > is to have udev's mdadm -i command run with watershed, which should i= n > theory serialize it. What do you think? >=20 > SUBSYSTEM=3D=3D"block", ACTION=3D=3D"add|change", ENV{ID_FS_TYPE}=3D=3D= "linux_raid*", \ > RUN+=3D"watershed -i mdadm /sbin/mdadm --incremental $env{DEVNAME}= " I haven't come across watershed before. I couldn't easily find out muc= h about it on the web, so I cannot say what effect it would have. My gue= ss from what little I have read is 'none'. >=20 > Finally, in your view, is it essential that the underlying partitions > used in the md's be the "Linux raid autodetect" type? My partitions a= t > the moment are just plain "Linux". I actually recommend "Non-FS data" (0xDA) as 'Linux' might make some to= ols think there is a filesystem there even though there isn't. But 'Linux'= is mostly fine. I avoid "Linux raid autodetect" as it enable the MD_AUTOD= ETECT functionality which I don't like. NeilBrown >=20 > Anyway, I will test mdadm 3.1.3 right now but I wanted to ask for you= r > insight/comments on the above. Thanks! >=20 > Best, > Tommy >=20 >=20 >=20 > On Sun, Aug 8, 2010 at 1:58 AM, Neil Brown wrote: > > On Sat, 7 Aug 2010 18:27:58 -0700 > > "fibreraid@gmail.com" wrote: > > > >> Hi all, > >> > >> I am facing a serious issue with md's on my Ubuntu 10.04 64-bit > >> server. I am using mdadm 3.1.2. The system has 40 drives in it, an= d > >> there are 10 md devices, which are a combination of RAID 0, 1, 5, = 6, > >> and 10 levels. The drives are connected via LSI SAS adapters in > >> external SAS JBODs. > >> > >> When I boot the system, about 50% of the time, the md's will not c= ome > >> up correctly. Instead of md0-md9 being active, some or all will be > >> inactive and there will be new md's like md127, md126, md125, etc. > > > > Sounds like a locking problem - udev is calling "mdadm -I" on each = device and > > might call some in parallel. =C2=A0mdadm needs to serialise things = to ensure this > > sort of confusion doesn't happen. > > > > It is possible that this is fixed in the just-released mdadm-3.1.3.= =C2=A0If you > > could test and and see if it makes a difference that would help a l= ot. > > > > Thanks, > > NeilBrown > > > >> > >> Here is the output of /proc/mdstat when all md's come up correctly= : > >> > >> > >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid= 5] > >> [raid4] [raid10] > >> md0 : active raid6 sdj1[6] sdk1[7] sdf1[2] sdb1[10] sdg1[3] sdl1[8= ](S) > >> sdh1[4] sdm1[9] sde1[1] sdi1[12](S) sdc1[11] sdd1[0] > >> =C2=A0 =C2=A0 =C2=A0 1146967040 blocks super 1.2 level 6, 128k chu= nk, algorithm 2 > >> [10/10] [UUUUUUUUUU] > >> > >> md9 : active raid0 sdao1[1] sdan1[0] > >> =C2=A0 =C2=A0 =C2=A0 976765440 blocks super 1.2 256k chunks > >> > >> md8 : active raid0 sdam1[1] sdal1[0] > >> =C2=A0 =C2=A0 =C2=A0 976765440 blocks super 1.2 256k chunks > >> > >> md7 : active raid0 sdak1[1] sdaj1[0] > >> =C2=A0 =C2=A0 =C2=A0 976765888 blocks super 1.2 4k chunks > >> > >> md6 : active raid0 sdai1[1] sdah1[0] > >> =C2=A0 =C2=A0 =C2=A0 976765696 blocks super 1.2 128k chunks > >> > >> md5 : active raid0 sdag1[1] sdaf1[0] > >> =C2=A0 =C2=A0 =C2=A0 976765440 blocks super 1.2 256k chunks > >> > >> md4 : active raid0 sdae1[1] sdad1[0] > >> =C2=A0 =C2=A0 =C2=A0 976765888 blocks super 1.2 32k chunks > >> > >> md3 : active raid1 sdac1[1] sdab1[0] > >> =C2=A0 =C2=A0 =C2=A0 195357272 blocks super 1.2 [2/2] [UU] > >> > >> md2 : active raid0 sdaa1[0] sdz1[1] > >> =C2=A0 =C2=A0 =C2=A0 62490672 blocks super 1.2 4k chunks > >> > >> md1 : active raid5 sdy1[10] sdx1[9] sdw1[8] sdv1[7] sdu1[6] sdt1[5= ] > >> sds1[4] sdr1[3] sdq1[2] sdp1[11](S) sdo1[1] sdn1[0] > >> =C2=A0 =C2=A0 =C2=A0 2929601120 blocks super 1.2 level 5, 16k chun= k, algorithm 2 > >> [11/11] [UUUUUUUUUUU] > >> > >> unused devices: > >> > >> > >> ------------------------------------------------------------------= -------------------------------------------------------- > >> > >> > >> Here are several examples of when they do not come up correctly. > >> Again, I am not making any configuration changes; I just reboot th= e > >> system and check /proc/mdstat several minutes after it is fully > >> booted. > >> > >> > >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid= 5] > >> [raid4] [raid10] > >> md124 : inactive sdam1[1](S) > >> =C2=A0 =C2=A0 =C2=A0 488382944 blocks super 1.2 > >> > >> md125 : inactive sdag1[1](S) > >> =C2=A0 =C2=A0 =C2=A0 488382944 blocks super 1.2 > >> > >> md7 : active raid0 sdaj1[0] sdak1[1] > >> =C2=A0 =C2=A0 =C2=A0 976765888 blocks super 1.2 4k chunks > >> > >> md126 : inactive sdw1[8](S) sdn1[0](S) sdo1[1](S) sdu1[6](S) > >> sdq1[2](S) sdx1[9](S) > >> =C2=A0 =C2=A0 =C2=A0 1757761512 blocks super 1.2 > >> > >> md9 : active raid0 sdan1[0] sdao1[1] > >> =C2=A0 =C2=A0 =C2=A0 976765440 blocks super 1.2 256k chunks > >> > >> md6 : inactive sdah1[0](S) > >> =C2=A0 =C2=A0 =C2=A0 488382944 blocks super 1.2 > >> > >> md4 : inactive sdae1[1](S) > >> =C2=A0 =C2=A0 =C2=A0 488382944 blocks super 1.2 > >> > >> md8 : inactive sdal1[0](S) > >> =C2=A0 =C2=A0 =C2=A0 488382944 blocks super 1.2 > >> > >> md127 : inactive sdg1[3](S) sdl1[8](S) sdc1[11](S) sdi1[12](S) > >> sdf1[2](S) sdb1[10](S) > >> =C2=A0 =C2=A0 =C2=A0 860226027 blocks super 1.2 > >> > >> md5 : inactive sdaf1[0](S) > >> =C2=A0 =C2=A0 =C2=A0 488382944 blocks super 1.2 > >> > >> md1 : inactive sdr1[3](S) sdp1[11](S) sdt1[5](S) sds1[4](S) > >> sdy1[10](S) sdv1[7](S) > >> =C2=A0 =C2=A0 =C2=A0 1757761512 blocks super 1.2 > >> > >> md0 : inactive sde1[1](S) sdh1[4](S) sdm1[9](S) sdj1[6](S) sdd1[0]= (S) sdk1[7](S) > >> =C2=A0 =C2=A0 =C2=A0 860226027 blocks super 1.2 > >> > >> md3 : inactive sdab1[0](S) > >> =C2=A0 =C2=A0 =C2=A0 195357344 blocks super 1.2 > >> > >> md2 : active raid0 sdaa1[0] sdz1[1] > >> =C2=A0 =C2=A0 =C2=A0 62490672 blocks super 1.2 4k chunks > >> > >> unused devices: > >> > >> > >> ------------------------------------------------------------------= --------------------------------------------------------- > >> > >> > >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid= 5] > >> [raid4] [raid10] > >> md126 : inactive sdaf1[0](S) > >> =C2=A0 =C2=A0 =C2=A0 488382944 blocks super 1.2 > >> > >> md127 : inactive sdae1[1](S) > >> =C2=A0 =C2=A0 =C2=A0 488382944 blocks super 1.2 > >> > >> md9 : active raid0 sdan1[0] sdao1[1] > >> =C2=A0 =C2=A0 =C2=A0 976765440 blocks super 1.2 256k chunks > >> > >> md7 : active raid0 sdaj1[0] sdak1[1] > >> =C2=A0 =C2=A0 =C2=A0 976765888 blocks super 1.2 4k chunks > >> > >> md4 : inactive sdad1[0](S) > >> =C2=A0 =C2=A0 =C2=A0 488382944 blocks super 1.2 > >> > >> md6 : active raid0 sdah1[0] sdai1[1] > >> =C2=A0 =C2=A0 =C2=A0 976765696 blocks super 1.2 128k chunks > >> > >> md8 : active raid0 sdam1[1] sdal1[0] > >> =C2=A0 =C2=A0 =C2=A0 976765440 blocks super 1.2 256k chunks > >> > >> md5 : inactive sdag1[1](S) > >> =C2=A0 =C2=A0 =C2=A0 488382944 blocks super 1.2 > >> > >> md0 : active raid6 sdc1[11] sdd1[0] sdh1[4] sdf1[2] sdm1[9] sde1[1= ] > >> sdb1[10] sdg1[3] sdl1[8](S) sdj1[6] sdk1[7] sdi1[12](S) > >> =C2=A0 =C2=A0 =C2=A0 1146967040 blocks super 1.2 level 6, 128k chu= nk, algorithm 2 > >> [10/10] [UUUUUUUUUU] > >> > >> md1 : active raid5 sdq1[2] sdy1[10] sdv1[7] sdn1[0] sdt1[5] sdw1[8= ] > >> sdp1[11](S) sdr1[3] sdu1[6] sdx1[9] sdo1[1] sds1[4] > >> =C2=A0 =C2=A0 =C2=A0 2929601120 blocks super 1.2 level 5, 16k chun= k, algorithm 2 > >> [11/11] [UUUUUUUUUUU] > >> > >> md3 : active raid1 sdac1[1] sdab1[0] > >> =C2=A0 =C2=A0 =C2=A0 195357272 blocks super 1.2 [2/2] [UU] > >> > >> md2 : active raid0 sdz1[1] sdaa1[0] > >> =C2=A0 =C2=A0 =C2=A0 62490672 blocks super 1.2 4k chunks > >> > >> unused devices: > >> > >> > >> ------------------------------------------------------------------= -------------------------------------------------------- > >> > >> > >> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid= 5] > >> [raid4] [raid10] > >> md127 : inactive sdab1[0](S) > >> =C2=A0 =C2=A0 =C2=A0 195357344 blocks super 1.2 > >> > >> md4 : active raid0 sdad1[0] sdae1[1] > >> =C2=A0 =C2=A0 =C2=A0 976765888 blocks super 1.2 32k chunks > >> > >> md7 : active raid0 sdak1[1] sdaj1[0] > >> =C2=A0 =C2=A0 =C2=A0 976765888 blocks super 1.2 4k chunks > >> > >> md8 : active raid0 sdam1[1] sdal1[0] > >> =C2=A0 =C2=A0 =C2=A0 976765440 blocks super 1.2 256k chunks > >> > >> md6 : active raid0 sdah1[0] sdai1[1] > >> =C2=A0 =C2=A0 =C2=A0 976765696 blocks super 1.2 128k chunks > >> > >> md9 : active raid0 sdao1[1] sdan1[0] > >> =C2=A0 =C2=A0 =C2=A0 976765440 blocks super 1.2 256k chunks > >> > >> md5 : active raid0 sdaf1[0] sdag1[1] > >> =C2=A0 =C2=A0 =C2=A0 976765440 blocks super 1.2 256k chunks > >> > >> md1 : active raid5 sdy1[10] sdv1[7] sdu1[6] sds1[4] sdq1[2] > >> sdp1[11](S) sdt1[5] sdo1[1] sdx1[9] sdr1[3] sdw1[8] sdn1[0] > >> =C2=A0 =C2=A0 =C2=A0 2929601120 blocks super 1.2 level 5, 16k chun= k, algorithm 2 > >> [11/11] [UUUUUUUUUUU] > >> > >> md0 : active raid6 sdl1[8](S) sdd1[0] sdc1[11] sdg1[3] sdk1[7] sde= 1[1] > >> sdm1[9] sdb1[10] sdi1[12](S) sdh1[4] sdf1[2] sdj1[6] > >> =C2=A0 =C2=A0 =C2=A0 1146967040 blocks super 1.2 level 6, 128k chu= nk, algorithm 2 > >> [10/10] [UUUUUUUUUU] > >> > >> md3 : inactive sdac1[1](S) > >> =C2=A0 =C2=A0 =C2=A0 195357344 blocks super 1.2 > >> > >> md2 : active raid0 sdz1[1] sdaa1[0] > >> =C2=A0 =C2=A0 =C2=A0 62490672 blocks super 1.2 4k chunks > >> > >> unused devices: > >> > >> > >> > >> My mdadm.conf file is as follows: > >> > >> > >> # mdadm.conf > >> # > >> # Please refer to mdadm.conf(5) for information about this file. > >> # > >> > >> # by default, scan all partitions (/proc/partitions) for MD superb= locks. > >> # alternatively, specify devices to scan, using wildcards if desir= ed. > >> DEVICE partitions > >> > >> # auto-create devices with Debian standard permissions > >> CREATE owner=3Droot group=3Ddisk mode=3D0660 auto=3Dyes > >> > >> # automatically tag new arrays as belonging to the local system > >> HOMEHOST > >> > >> # instruct the monitoring daemon where to send mail alerts > >> MAILADDR root > >> > >> # definitions of existing MD arrays > >> > >> # This file was auto-generated on Sun, 13 Jul 2008 20:42:57 -0500 > >> # by mkconf $Id$ > >> > >> > >> > >> > >> Any insight would be greatly appreciated. This is a big problem as= it > >> is now. Thank you very much in advance! > >> > >> Best, > >> -Tommy > >> -- > >> To unsubscribe from this list: send the line "unsubscribe linux-ra= id" in > >> the body of a message to majordomo@vger.kernel.org > >> More majordomo info at =C2=A0http://vger.kernel.org/majordomo-info= =2Ehtml > > > > -- To unsubscribe from this list: send the line "unsubscribe linux-raid" i= n the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html