* Wierd critical node problem @ 2008-06-10 7:24 Wayne Gemmell 2008-06-10 8:15 ` NeilBrown 0 siblings, 1 reply; 6+ messages in thread From: Wayne Gemmell @ 2008-06-10 7:24 UTC (permalink / raw) To: linux-raid Hi all I run a 4 disk array with 6 RAID 1 partitions. This is a very simple configurations which has served me well through multiple drive failures. I now have a wierd problem. One of the disks has become a critical node. My server does not boot without this disk no matter which disk is my primary boot disk (bios). I've changed the order of the disks and it makes no difference. When this disk is not in booting from some of the other disks gives me a warning that there is an invalid superblock magic on sda. They all end up with an error on stdin and modprobe failing. Any ideas how to resolve this? I'm leaning towards nukeing sd[acd] and re-adding them. -- Regards Wayne ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Wierd critical node problem 2008-06-10 7:24 Wierd critical node problem Wayne Gemmell @ 2008-06-10 8:15 ` NeilBrown 2008-06-10 8:30 ` Wayne Gemmell 0 siblings, 1 reply; 6+ messages in thread From: NeilBrown @ 2008-06-10 8:15 UTC (permalink / raw) To: wayne; +Cc: linux-raid On Tue, June 10, 2008 5:24 pm, Wayne Gemmell wrote: > Hi all > > I run a 4 disk array with 6 RAID 1 partitions. This is a very simple > configurations which has served me well through multiple drive failures. I > now have a wierd problem. One of the disks has become a critical node. My > server does not boot without this disk no matter which disk is my primary > boot disk (bios). I've changed the order of the disks and it makes no > difference. > > When this disk is not in booting from some of the other disks gives me a > warning that there is an invalid superblock magic on sda. They all end up > with an error on stdin and modprobe failing. > > > Any ideas how to resolve this? I'm leaning towards nukeing sd[acd] and > re-adding them. I suggest you provide lots more details. Probably mdadm -Dsv and mdadm -Esv would be a good start. NeilBrown ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Wierd critical node problem 2008-06-10 8:15 ` NeilBrown @ 2008-06-10 8:30 ` Wayne Gemmell 2008-06-10 10:06 ` NeilBrown 0 siblings, 1 reply; 6+ messages in thread From: Wayne Gemmell @ 2008-06-10 8:30 UTC (permalink / raw) To: linux-raid; +Cc: NeilBrown Sure thingk. On Tuesday 10 June 2008 10:15:22 you wrote: > I suggest you provide lots more details. > Probably > mdadm -Dsv ARRAY /dev/md5 level=raid1 num-devices=2 UUID=6868a02e:0e985748:cae821e2:1cf91e6d devices=/dev/sdc2,/dev/sdd2 ARRAY /dev/md0 level=raid1 num-devices=4 UUID=222850cc:3ee166b9:9e71a84f:e86d40a1 devices=/dev/sdb1,/dev/sda1,/dev/sdd1,/dev/sdc1 ARRAY /dev/md6 level=raid1 num-devices=2 UUID=bfa44b7a:d6a2d5fc:cae821e2:1cf91e6d devices=/dev/sdb2,/dev/sda2 ARRAY /dev/md1 level=raid1 num-devices=4 UUID=5f6e694f:1d5441a3:7c5e3c07:b6a0267e devices=/dev/sdb3,/dev/sdd3,/dev/sdc3,/dev/sda3 ARRAY /dev/md2 level=raid1 num-devices=4 UUID=020b3642:b32a5fda:ebae4acf:da43fee2 devices=/dev/sdc5,/dev/sdd5,/dev/sda5,/dev/sdb5 ARRAY /dev/md3 level=raid1 num-devices=4 UUID=da7d0fc4:ba7f7bd0:bfe35e68:eec9a5cb devices=/dev/sda6,/dev/sdc6,/dev/sdd6,/dev/sdb6 ARRAY /dev/md4 level=raid1 num-devices=4 UUID=5785dcb6:10ba80a4:b169e59f:d80bc484 devices=/dev/sdc7,/dev/sda7,/dev/sdb7,/dev/sdd7 > and > mdadm -Esv > would be a good start. ARRAY /dev/md6 level=raid1 num-devices=2 UUID=bfa44b7a:d6a2d5fc:cae821e2:1cf91e6d devices=/dev/sdb2,/dev/sda2 ARRAY /dev/md0 level=raid1 num-devices=4 UUID=222850cc:3ee166b9:9e71a84f:e86d40a1 devices=/dev/sdd1,/dev/sdc1,/dev/sdb1,/dev/sda1 ARRAY /dev/md5 level=raid1 num-devices=2 UUID=6868a02e:0e985748:cae821e2:1cf91e6d devices=/dev/sdd2,/dev/sdc2 ARRAY /dev/md1 level=raid1 num-devices=4 UUID=5f6e694f:1d5441a3:7c5e3c07:b6a0267e devices=/dev/sdd3,/dev/sdc3,/dev/sdb3,/dev/sda3 ARRAY /dev/md2 level=raid1 num-devices=4 UUID=020b3642:b32a5fda:ebae4acf:da43fee2 devices=/dev/sdd5,/dev/sdc5,/dev/sdb5,/dev/sda5 ARRAY /dev/md3 level=raid1 num-devices=4 UUID=da7d0fc4:ba7f7bd0:bfe35e68:eec9a5cb devices=/dev/sdd6,/dev/sdc6,/dev/sdb6,/dev/sda6 ARRAY /dev/md4 level=raid1 num-devices=4 UUID=5785dcb6:10ba80a4:b169e59f:d80bc484 devices=/dev/sdd7,/dev/sdc7,/dev/sdb7,/dev/sda7 I have found the following in my logs, Jun 9 17:01:50 lloyd kernel: [ 52.912904] mdadm[2656]: segfault at 0000000000000004 rip 000000000041724c rsp 00007ffff99d9b30 error 4 -- Regards Wayne ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Wierd critical node problem 2008-06-10 8:30 ` Wayne Gemmell @ 2008-06-10 10:06 ` NeilBrown 2008-06-10 10:34 ` Wayne Gemmell 0 siblings, 1 reply; 6+ messages in thread From: NeilBrown @ 2008-06-10 10:06 UTC (permalink / raw) To: wayne; +Cc: linux-raid On Tue, June 10, 2008 6:30 pm, Wayne Gemmell wrote: > Sure thingk. > > On Tuesday 10 June 2008 10:15:22 you wrote: >> I suggest you provide lots more details. >> Probably >> mdadm -Dsv Damn, I meant to say "-Dsvv" (2 v's) but it doens't really matter, I think that is a good enough picture. However.... > I have found the following in my logs, > Jun 9 17:01:50 lloyd kernel: [ 52.912904] mdadm[2656]: segfault at > 0000000000000004 rip 000000000041724c rsp 00007ffff99d9b30 error 4 > I suspect this is the real problem. Which version of mdadm (mdadm -V)? If these arrays are being assembled by the initrd, you would need to find out what mdadm is in the initrd, though it is probably the same as in /sbin. What distro. What kernel version? NeilBrown ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Wierd critical node problem 2008-06-10 10:06 ` NeilBrown @ 2008-06-10 10:34 ` Wayne Gemmell 2008-06-19 5:07 ` Neil Brown 0 siblings, 1 reply; 6+ messages in thread From: Wayne Gemmell @ 2008-06-10 10:34 UTC (permalink / raw) To: NeilBrown; +Cc: linux-raid On Tuesday 10 June 2008 12:06:10 NeilBrown wrote: > On Tue, June 10, 2008 6:30 pm, Wayne Gemmell wrote: > > Sure thingk. > > > > On Tuesday 10 June 2008 10:15:22 you wrote: > >> I suggest you provide lots more details. > >> Probably > >> mdadm -Dsv > > Damn, I meant to say "-Dsvv" (2 v's) but it doens't really matter, > I think that is a good enough picture. > However.... Just for thouroughness.... /dev/md5: Version : 00.90.03 Creation Time : Mon Jul 30 13:41:01 2007 Raid Level : raid1 Array Size : 979840 (957.04 MiB 1003.36 MB) Used Dev Size : 979840 (957.04 MiB 1003.36 MB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 5 Persistence : Superblock is persistent Update Time : Tue Jun 10 10:08:45 2008 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 UUID : 6868a02e:0e985748:cae821e2:1cf91e6d (local to host lloyd) Events : 0.84 Number Major Minor RaidDevice State 0 8 34 0 active sync /dev/sdc2 1 8 50 1 active sync /dev/sdd2 /dev/md0: Version : 00.90.03 Creation Time : Wed Aug 30 08:17:25 2006 Raid Level : raid1 Array Size : 489856 (478.46 MiB 501.61 MB) Used Dev Size : 489856 (478.46 MiB 501.61 MB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Mon Jun 9 18:02:55 2008 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 UUID : 222850cc:3ee166b9:9e71a84f:e86d40a1 Events : 0.1598 Number Major Minor RaidDevice State 0 8 17 0 active sync /dev/sdb1 1 8 1 1 active sync /dev/sda1 2 8 49 2 active sync /dev/sdd1 3 8 33 3 active sync /dev/sdc1 /dev/md6: Version : 00.90.03 Creation Time : Mon Jul 30 13:46:12 2007 Raid Level : raid1 Array Size : 979840 (957.04 MiB 1003.36 MB) Used Dev Size : 979840 (957.04 MiB 1003.36 MB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 6 Persistence : Superblock is persistent Update Time : Tue Jun 10 10:08:46 2008 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 UUID : bfa44b7a:d6a2d5fc:cae821e2:1cf91e6d (local to host lloyd) Events : 0.5980 Number Major Minor RaidDevice State 0 8 18 0 active sync /dev/sdb2 1 8 2 1 active sync /dev/sda2 /dev/md1: Version : 00.90.03 Creation Time : Wed Aug 30 08:18:01 2006 Raid Level : raid1 Array Size : 4883648 (4.66 GiB 5.00 GB) Used Dev Size : 4883648 (4.66 GiB 5.00 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 1 Persistence : Superblock is persistent Update Time : Tue Jun 10 12:24:23 2008 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 UUID : 5f6e694f:1d5441a3:7c5e3c07:b6a0267e Events : 0.17895140 Number Major Minor RaidDevice State 0 8 19 0 active sync /dev/sdb3 1 8 51 1 active sync /dev/sdd3 2 8 35 2 active sync /dev/sdc3 3 8 3 3 active sync /dev/sda3 /dev/md2: Version : 00.90.03 Creation Time : Wed Aug 30 08:18:46 2006 Raid Level : raid1 Array Size : 9767424 (9.31 GiB 10.00 GB) Used Dev Size : 9767424 (9.31 GiB 10.00 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 2 Persistence : Superblock is persistent Update Time : Tue Jun 10 12:24:08 2008 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 UUID : 020b3642:b32a5fda:ebae4acf:da43fee2 Events : 0.15926826 Number Major Minor RaidDevice State 0 8 37 0 active sync /dev/sdc5 1 8 53 1 active sync /dev/sdd5 2 8 5 2 active sync /dev/sda5 3 8 21 3 active sync /dev/sdb5 /dev/md3: Version : 00.90.03 Creation Time : Wed Aug 30 08:19:37 2006 Raid Level : raid1 Array Size : 1951744 (1906.32 MiB 1998.59 MB) Used Dev Size : 1951744 (1906.32 MiB 1998.59 MB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 3 Persistence : Superblock is persistent Update Time : Tue Jun 10 12:24:03 2008 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 UUID : da7d0fc4:ba7f7bd0:bfe35e68:eec9a5cb Events : 0.1515534 Number Major Minor RaidDevice State 0 8 6 0 active sync /dev/sda6 1 8 38 1 active sync /dev/sdc6 2 8 54 2 active sync /dev/sdd6 3 8 22 3 active sync /dev/sdb6 /dev/md4: Version : 00.90.03 Creation Time : Fri Sep 15 11:20:47 2006 Raid Level : raid1 Array Size : 138215104 (131.81 GiB 141.53 GB) Used Dev Size : 138215104 (131.81 GiB 141.53 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 4 Persistence : Superblock is persistent Update Time : Tue Jun 10 12:24:27 2008 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 UUID : 5785dcb6:10ba80a4:b169e59f:d80bc484 Events : 0.28934180 Number Major Minor RaidDevice State 0 8 39 0 active sync /dev/sdc7 1 8 7 1 active sync /dev/sda7 2 8 23 2 active sync /dev/sdb7 3 8 55 3 active sync /dev/sdd7 > > > I have found the following in my logs, > > Jun 9 17:01:50 lloyd kernel: [ 52.912904] mdadm[2656]: segfault at > > 0000000000000004 rip 000000000041724c rsp 00007ffff99d9b30 error 4 > > I suspect this is the real problem. > Which version of mdadm (mdadm -V)? mdadm - v2.6.2 - 21st May 2007 > > If these arrays are being assembled by the initrd, you would need > to find out what mdadm is in the initrd, though it is probably > the same as in /sbin. What distro. What kernel version? I'm running Ubuntu Gutsy running 2.6.22-14-server kernel. I May have an old version of mdadm in initrd so I've regenerated it now. I'll only really get to test it again tomorrow. -- Regards Wayne ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Wierd critical node problem 2008-06-10 10:34 ` Wayne Gemmell @ 2008-06-19 5:07 ` Neil Brown 0 siblings, 0 replies; 6+ messages in thread From: Neil Brown @ 2008-06-19 5:07 UTC (permalink / raw) To: wayne; +Cc: linux-raid On Tuesday June 10, wayne@flashmedia.co.za wrote: > > > > > I have found the following in my logs, > > > Jun 9 17:01:50 lloyd kernel: [ 52.912904] mdadm[2656]: segfault at > > > 0000000000000004 rip 000000000041724c rsp 00007ffff99d9b30 error 4 > > > > I suspect this is the real problem. > > Which version of mdadm (mdadm -V)? > mdadm - v2.6.2 - 21st May 2007 > I don't know of any segfault problems with this version. It might be worth trying a newer release: 2.6.4 or 2.6.7. If you are still having problems, can you get the exact text of the error messages, they might show something useful. NeilBrown ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2008-06-19 5:07 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2008-06-10 7:24 Wierd critical node problem Wayne Gemmell 2008-06-10 8:15 ` NeilBrown 2008-06-10 8:30 ` Wayne Gemmell 2008-06-10 10:06 ` NeilBrown 2008-06-10 10:34 ` Wayne Gemmell 2008-06-19 5:07 ` Neil Brown
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).