* System hangs on raid md recovery/resync [not found] <da290bd50807260519v293851c9ue4bef88b1fc52013@mail.gmail.com> @ 2008-07-27 22:05 ` Brad 2008-07-27 22:21 ` Roger Heflin 0 siblings, 1 reply; 6+ messages in thread From: Brad @ 2008-07-27 22:05 UTC (permalink / raw) To: linux-raid Hi. I'm running Linux 2.6.26 with mdadm v2.6.1. Over the past 24 hours I've several times set up a 400GB raid1 md array in a recovery/resync operation which has subsequently hung the system. In five such operations three have hung: o I added a third disk drive to a working raid1 md device; after an hour or more of active synchronisation the system hung. o after pulling out the third (hot pluggable) disk I rebooted the system, which started resyncing the md device upon assembly. This operation also hung after about an hour. o rebooting again and this time reducing all activity on the system to an absolute minimum the resync succeeded. o I tried again to mirror the md device to my third hot-pluggable disk by inserting the drive and attaching it to the raid1 md device; after an hour or so the recovery hung again. o rebooting again with the third drive unplugged it looks like the resync is going to run to completion this time. All three disks are Western Digital SATA 2 drives. SMART says there's no problems with the drives. A resync/recover operation typically proceeds at an average speed of about 35MB/sec, as reported by /proc/mdstat. But then - for the times that it hung - /proc/mdstat reports slower and slower speeds and longer and longer finish times (30,000 minutes plus!). In /sys/block/md1/md the value of sync_completed would stay static and sync_speed would drop lower and lower (< 1000KB/sec). I tried: echo 40960 > sync_speed_min in an attempt to try and coax things to go faster but the system remained hung. The system was hung in that: o load average increased to about 13; top reported 50% spent in 'wait time'; o Any operation that accessed the disk/md device would 'hang'. Other trivial operations - shell builtin commands, X11 widget updates - still worked. 'shutdown -r now' wouldn't work; I had to cold-boot the system each time. o No error messages logged to the console or syslog. This 'hang' *seems* to be related to system activity; the system has never been *heavily* loaded the three times a resync/recover operation failed but I had a couple of download programs and the like - keeping the network interface mildly busy - running in every failed/hung case. Ideally the resync/recover operation should proceed independent of the system activity, I would have thought? I'd hoped to be able to perform daily/weekly transparent backups by plugging in the third drive, adding it to the raid1 md device and then detaching the disk after the recover operation had completed. Can anyone help? I have no idea if there are other things I can do or tune to get around this problem, or if it's an actual bug. I had a look in the kernel archives but couldn't see anything that seemed relevant to this problem with the latest stable kernel. Thanks, Brad ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: System hangs on raid md recovery/resync 2008-07-27 22:05 ` System hangs on raid md recovery/resync Brad @ 2008-07-27 22:21 ` Roger Heflin 2008-07-27 23:39 ` Brad 0 siblings, 1 reply; 6+ messages in thread From: Roger Heflin @ 2008-07-27 22:21 UTC (permalink / raw) To: Brad; +Cc: linux-raid Brad wrote: > Hi. I'm running Linux 2.6.26 with mdadm v2.6.1. Over the past 24 > hours I've several times set up a 400GB raid1 md array in a > recovery/resync operation which has subsequently hung the system. > In five such operations three have hung: > > o I added a third disk drive to a working raid1 md device; after > an hour or more of active synchronisation the system hung. > > o after pulling out the third (hot pluggable) disk I rebooted the > system, which started resyncing the md device upon assembly. > This operation also hung after about an hour. > > o rebooting again and this time reducing all activity on the system > to an absolute minimum the resync succeeded. > > o I tried again to mirror the md device to my third hot-pluggable > disk by inserting the drive and attaching it to the raid1 md device; > after an hour or so the recovery hung again. > > o rebooting again with the third drive unplugged it looks like > the resync is going to run to completion this time. > > All three disks are Western Digital SATA 2 drives. SMART says > there's no problems with the drives. > > A resync/recover operation typically proceeds at an average > speed of about 35MB/sec, as reported by /proc/mdstat. But > then - for the times that it hung - /proc/mdstat reports slower > and slower speeds and longer and longer finish times (30,000 minutes > plus!). In /sys/block/md1/md the value of sync_completed > would stay static and sync_speed would drop lower and lower > (< 1000KB/sec). > > I tried: > > echo 40960 > sync_speed_min > > in an attempt to try and coax things to go faster but the system > remained hung. If the hardware setup cannot do faster than 35 MB/second nothing you can do will make it go faster. > > The system was hung in that: > > o load average increased to about 13; top reported 50% spent > in 'wait time'; > > o Any operation that accessed the disk/md device would 'hang'. > Other trivial operations - shell builtin commands, X11 widget > updates - still worked. 'shutdown -r now' wouldn't work; I had > to cold-boot the system each time. > > o No error messages logged to the console or syslog. > > This 'hang' *seems* to be related to system activity; the system > has never been *heavily* loaded the three times a resync/recover > operation failed but I had a couple of download programs and the > like - keeping the network interface mildly busy - running in every > failed/hung case. > > Ideally the resync/recover operation should proceed independent > of the system activity, I would have thought? I'd hoped to be able to > perform daily/weekly transparent backups by plugging in the third drive, > adding it to the raid1 md device and then detaching the disk > after the recover operation had completed. > > Can anyone help? I have no idea if there are other things I can > do or tune to get around this problem, or if it's an actual bug. I had > a look in the kernel archives but couldn't see anything that seemed > relevant to this problem with the latest stable kernel. > > Thanks, It really sounds like a HW issue of some sort, a weak power supply, or a badly designed MB. I have a badly designed one here that if I put disks on certain built-in MB ports then things work just fine for weeks and weeks until something happens to cause a resync, and on any resync the MTBF is <30 minutes, moving the exact same disks off of those built-in MB ports makes things work just fine, and having the disk on the built-in MB ports also results in other cards in the PCI bus having serious issues (losing data and other crap). What kind of motherboard is it and which chipset is on that MB, and what kind of sata ports are the disks on? If you can get all 3 disks in the machine and working (without a resync) then I would try doing "dd if=/dev/sda of=/dev/null bs=1M" for each of the 3 disks at the same time and see if that causes a hang, if it does it is not a MD issue, also I would check the speed of 1 disk, 2 disks, and 3 disks and see how bad those ports bottleneck with multiple disks being used. Roger ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: System hangs on raid md recovery/resync 2008-07-27 22:21 ` Roger Heflin @ 2008-07-27 23:39 ` Brad 2008-07-28 0:08 ` Roger Heflin 0 siblings, 1 reply; 6+ messages in thread From: Brad @ 2008-07-27 23:39 UTC (permalink / raw) To: Roger Heflin; +Cc: linux-raid Roger, thanks for your reply. > If the hardware setup cannot do faster than 35 MB/second nothing you can do > will make it go faster. On a straight read of any of the disks, using dd or 'hdparm -t', I can get about 70MB/sec to 75MB/sec out of the disk drives. I've always thought that the MD software deliberately 'throttled' a resync/recover operation, slowing things down if the disks were being used by applications, so I've been happy with around 35MB/sec for my resyncs, as the system has usually been lightly loaded with other programs using the drives. > What kind of motherboard is it and which chipset is on that MB, and what > kind of sata ports are the disks on? > > If you can get all 3 disks in the machine and working (without a resync) > then I would try doing "dd if=/dev/sda of=/dev/null bs=1M" for each of the 3 > disks at the same time and see if that causes a hang, if it does it is not a > MD issue, also I would check the speed of 1 disk, 2 disks, and 3 disks and > see how bad those ports bottleneck with multiple disks being used. My system is 7 months old; the motherboard is a Gigabyte GA-P35-DS4. It has an Intel ICH9R northbridge with 6 SATA 2 ports and a 'Gigabyte' (JMicron 20360/20363) southbridge with 2 SATA 2 ports. I have two 500GB Western Digital SATA 2 internal disks, one on each controller, in an MD raid1 mirror. I've experienced these problems while plugging in a third Western Digital 500GB drive into the ICH9R controller and adding it as a third mirror element to the raid1 MD device. Other than this 'hang' problem with MD I've never had a problem with any of the disks. For example, after yesterday failing to synchronise the third disk with the MD raid1 device, I proceeded to do a filesystem-level copy, using cpio to copy all the files from the MD device to the (separately mounted) third disk. That worked fine (took a lot longer, though, because of the huge number of small files I have on the filesystem). A follow-up rsync to 'catch' files that had been modified during the cpio also succeeded. I just now ran a dd test as you suggested of each disk, and each ran fine, with dd reporting speeds of 66.1, 68.6 and 70.5 MB/s. One 'hard resetting link' error/event was logged for one of the three SATA 2 ports without the dd process for that link seeing any error. I saw absolutely no such errors logged at all with my 'hang' problems in synchronising the raid1 device yesterday. Everything would proceed fine until the resync operation simply stopped - with /sys/block/md1/md/sync_completed static, showing no further progress - and the system then 'hanging' on anything that tried to access a disk. Brad ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: System hangs on raid md recovery/resync 2008-07-27 23:39 ` Brad @ 2008-07-28 0:08 ` Roger Heflin 2008-07-28 10:05 ` Brad 0 siblings, 1 reply; 6+ messages in thread From: Roger Heflin @ 2008-07-28 0:08 UTC (permalink / raw) To: Brad; +Cc: linux-raid Brad wrote: > Roger, thanks for your reply. > > My system is 7 months old; the motherboard is a Gigabyte GA-P35-DS4. > It has an Intel ICH9R northbridge with 6 SATA 2 ports and a 'Gigabyte' > (JMicron 20360/20363) southbridge with 2 SATA 2 ports. I have two > 500GB Western Digital SATA 2 internal disks, one on each controller, in > an MD raid1 mirror. I've experienced these problems while plugging in > a third Western Digital 500GB drive into the ICH9R controller and > adding it as a third mirror element to the raid1 MD device. > > Other than this 'hang' problem with MD I've never had a problem with > any of the disks. For example, after yesterday failing to synchronise the third > disk with the MD raid1 device, I proceeded to do a filesystem-level copy, using > cpio to copy all the files from the MD device to the (separately mounted) third > disk. That worked fine (took a lot longer, though, because of the huge number > of small files I have on the filesystem). A follow-up rsync to > 'catch' files that > had been modified during the cpio also succeeded. > > I just now ran a dd test as you suggested of each disk, and each ran fine, with > dd reporting speeds of 66.1, 68.6 and 70.5 MB/s. One 'hard resetting link' > error/event was logged for one of the three SATA 2 ports without the dd process > for that link seeing any error. I saw absolutely no such errors > logged at all with > my 'hang' problems in synchronising the raid1 device yesterday. Everything > would proceed fine until the resync operation simply stopped - with > /sys/block/md1/md/sync_completed static, showing no further > progress - and the system then 'hanging' on anything that tried > to access a disk. The Intel stuff tends to be pretty decent, where I have ran into the most issues is with anything that the MB vendor adds on, so I would try putting all 3 on the Intel, and in the past the Intel controllers I have tested have been able to run all disks at full speed (or close to it) even when multiple disks are being actively used, this would at least eliminate the jmicron controller from the mix. The sync will slow down quite a bit if other things are causing reads/writes that are causing the disks to seek around. How much power does the PS have on the 12V line? So long as it is either a split 12V supply or has more than 15-20A (non-split PS) you should be OK. I have had issues with non-split PS's that only had 15A on the 12V resulting in odd happenings with the disk. I would also setup the sysrq keys and the next time it happens do a "alt-sysrq" with a "T" "W" and a "Q" to test sysrq and make sure it is working do a "alt-sysrq-S" (sync) as this should produces messages in dmesg and show that things are enabled and working. You did run the dd on all 3 disks at the same time? The hard resetting link usually indicates something bad happened, though that could be caused by a lot of things. Roger ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: System hangs on raid md recovery/resync 2008-07-28 0:08 ` Roger Heflin @ 2008-07-28 10:05 ` Brad 2008-07-28 11:16 ` Roger Heflin 0 siblings, 1 reply; 6+ messages in thread From: Brad @ 2008-07-28 10:05 UTC (permalink / raw) To: Roger Heflin; +Cc: linux-raid Roger, > The Intel stuff tends to be pretty decent, where I have ran into the most > issues is with anything that the MB vendor adds on, so I would try putting > all 3 on the Intel, and in the past the Intel controllers I have tested have > been able to run all disks at full speed (or close to it) even when multiple > disks are being actively used, this would at least eliminate the jmicron > controller from the mix. I thought I was being cute in putting the two disks of the regular raid1 set on different controllers, for maximum redundancy. :-) But yes, the JMicron controller seems to be a bit flakey ... sometimes after a reboot it will experience constant 'hard resetting link' errors under full load. After another reboot though it'll be as stable as a rock. I've assumed Linux isn't getting something quite right when it initialises the JMicron controller at boot. In any case I've noticed - using iostat - that when I've added the third drive to the raid1 device, for example, the recover operation consists of reads on the disk that's hooked up to the ICH9R controller and writes to the third disk on the same controller. The MD code doesn't seem to share the read operation between the two existing mirrored disks, so the second disk on the JMicron isn't involved at all. > How much power does the PS have on the 12V line? So long as it is either > a split 12V supply or has more than 15-20A (non-split PS) you should be OK. I'm sorry but I wouldn't have a clue about that. I've got an Antec Sonata III 500 case with its standard 500W power suppler. "80 PLUS" certified, whatever that means ... "an EarthWatts 500 Watt power supply unit (PSU) which is equipped with universal input and Active PFC. This PSU is also 80PLUS(R) certified making it one of the most efficient PSU's available" says their web site. > You did run the dd on all 3 disks at the same time? Yes. > The hard resetting link usually indicates something bad happened, though > that could be caused by a lot of things. I normally never see that error at all. And I didn't see it the three times the system hung on the MD raid1 resync/recover operations. If the kernel MD code had a hardware problem with the SATA 2 ports would I see an error message? Brad ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: System hangs on raid md recovery/resync 2008-07-28 10:05 ` Brad @ 2008-07-28 11:16 ` Roger Heflin 0 siblings, 0 replies; 6+ messages in thread From: Roger Heflin @ 2008-07-28 11:16 UTC (permalink / raw) To: Brad; +Cc: linux-raid Brad wrote: > Roger, > >> The Intel stuff tends to be pretty decent, where I have ran into the most >> issues is with anything that the MB vendor adds on, so I would try putting >> all 3 on the Intel, and in the past the Intel controllers I have tested have >> been able to run all disks at full speed (or close to it) even when multiple >> disks are being actively used, this would at least eliminate the jmicron >> controller from the mix. > > I thought I was being cute in putting the two disks of the regular raid1 > set on different controllers, for maximum redundancy. :-) But yes, the > JMicron controller seems to be a bit flakey ... sometimes after a reboot it will > experience constant 'hard resetting link' errors under full load. After another > reboot though it'll be as stable as a rock. I've assumed Linux isn't getting > something quite right when it initialises the JMicron controller at boot. > > In any case I've noticed - using iostat - that when I've added the third drive > to the raid1 device, for example, the recover operation consists of reads on > the disk that's hooked up to the ICH9R controller and writes to the third > disk on the same controller. The MD code doesn't seem to share the read > operation between the two existing mirrored disks, so the second disk on > the JMicron isn't involved at all. From a performance stand point and if everything works fine it is a good idea, from a debugging and potential bug stand point, more drivers/more separate hardware involves more code/hardware so there is more stuff to have a bug/design defect. With everything being on the Intel sata controllers I would not expect it to make a performance difference either way. It could be the other traffic hitting the jmicron causes the failure when the Intel one is busy. On mine the issues seems to be that the MB system does not properly deal with things being heavily utilized (and I have seen at least 2-3 other MB's that as designed only fail when pushed close to the bandwidth limit of the internal buses), if things are not pushed too hard it will run fine for weeks with no errors, push it hard enough the right way (and in my case a raid5 MD rebuild is what does it) and you can make it fall over in just minutes with any other traffic outside the MD subsystem also needing to be there. > >> How much power does the PS have on the 12V line? So long as it is either >> a split 12V supply or has more than 15-20A (non-split PS) you should be OK. > > I'm sorry but I wouldn't have a clue about that. I've got an Antec > Sonata III 500 > case with its standard 500W power suppler. "80 PLUS" certified, whatever > that means ... "an EarthWatts 500 Watt power supply unit (PSU) which is > equipped with universal input and Active PFC. This PSU is also 80PLUS(R) > certified making it one of the most efficient PSU's available" says their web > site. I have a earthwatts 380, the 500 should be just fine, that is what is called a split rail supply, it has 2 separate 12V supplies. The ratings would be on the PS box itself, and on the PS itself. My 380 says 12V(v1) 17A and 12V(v2) 17A, a non-split rail supply will have only one 12V entry and it may be under 17A on the single supply. > >> You did run the dd on all 3 disks at the same time? > > Yes. > >> The hard resetting link usually indicates something bad happened, though >> that could be caused by a lot of things. > > I normally never see that error at all. And I didn't see it the three times the > system hung on the MD raid1 resync/recover operations. > > If the kernel MD code had a hardware problem with the SATA 2 ports would > I see an error message? The message would go on the text console, if it hangs the system bad enough it would likely never make it to the log files as the disk subsystem is hung, so it would not be in the log files. Also if you can switch to the console after the event the message would not be there, you would need to already have the text console up when the error happens. dmesg may have the message if enough of the system is still working for dmesg to run. Roger ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2008-07-28 11:16 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <da290bd50807260519v293851c9ue4bef88b1fc52013@mail.gmail.com>
2008-07-27 22:05 ` System hangs on raid md recovery/resync Brad
2008-07-27 22:21 ` Roger Heflin
2008-07-27 23:39 ` Brad
2008-07-28 0:08 ` Roger Heflin
2008-07-28 10:05 ` Brad
2008-07-28 11:16 ` Roger Heflin
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).