System hangs on raid md recovery/resync

System hangs on raid md recovery/resync

[Brad]

Hi.  I'm running Linux 2.6.26 with mdadm v2.6.1.  Over the past 24
hours I've several times set up a 400GB raid1 md array in a
recovery/resync operation which has subsequently hung the system.
In five such operations three have hung:

o  I added a third disk drive to a working raid1 md device; after
an hour or more of active synchronisation the system hung.

o  after pulling out the third (hot pluggable) disk I rebooted the
system, which started resyncing the md device upon assembly.
This operation also hung after about an hour.

o  rebooting again and this time reducing all activity on the system
to an absolute minimum the resync succeeded.

o  I tried again to mirror the md device to my third hot-pluggable
disk by inserting the drive and attaching it to the raid1 md device;
after an hour or so the recovery hung again.

o  rebooting again with the third drive unplugged it looks like
the resync is going to run to completion this time.

All three disks are Western Digital SATA 2 drives.  SMART says
there's no problems with the drives.

A resync/recover operation typically proceeds at an average
speed of about 35MB/sec, as reported by /proc/mdstat.  But
then - for the times that it hung - /proc/mdstat reports slower
and slower speeds and longer and longer finish times (30,000 minutes
plus!).  In /sys/block/md1/md the value of sync_completed
would stay static and sync_speed would drop lower and lower
(< 1000KB/sec).

I tried:

 echo 40960 > sync_speed_min

in an attempt to try and coax things to go faster but the system
remained hung.

The system was hung in that:

o  load average increased to about 13; top reported 50% spent
in 'wait time';

o  Any operation that accessed the disk/md device would 'hang'.
Other trivial operations - shell builtin commands, X11 widget
updates - still worked.  'shutdown -r now' wouldn't work; I had
to cold-boot the system each time.

o  No error messages logged to the console or syslog.

This 'hang' *seems* to be related to system activity; the system
has never been *heavily* loaded the three times a resync/recover
operation failed but I had a couple of download programs and the
like - keeping the network interface mildly busy - running in every
failed/hung case.

Ideally the resync/recover operation should proceed independent
of the system activity, I would have thought?  I'd hoped to be able to
perform daily/weekly transparent backups by plugging in the third drive,
adding it to the raid1 md device and then detaching the disk
after the recover operation had completed.

Can anyone help?  I have no idea if there are other things I can
do or tune to get around this problem, or if it's an actual bug.  I had
a look in the kernel archives but couldn't see anything that seemed
relevant to this problem with the latest stable kernel.

Thanks,


Brad

Re: System hangs on raid md recovery/resync

[Roger Heflin]

Brad wrote:
> Hi.  I'm running Linux 2.6.26 with mdadm v2.6.1.  Over the past 24
> hours I've several times set up a 400GB raid1 md array in a
> recovery/resync operation which has subsequently hung the system.
> In five such operations three have hung:
> 
> o  I added a third disk drive to a working raid1 md device; after
> an hour or more of active synchronisation the system hung.
> 
> o  after pulling out the third (hot pluggable) disk I rebooted the
> system, which started resyncing the md device upon assembly.
> This operation also hung after about an hour.
> 
> o  rebooting again and this time reducing all activity on the system
> to an absolute minimum the resync succeeded.
> 
> o  I tried again to mirror the md device to my third hot-pluggable
> disk by inserting the drive and attaching it to the raid1 md device;
> after an hour or so the recovery hung again.
> 
> o  rebooting again with the third drive unplugged it looks like
> the resync is going to run to completion this time.
> 
> All three disks are Western Digital SATA 2 drives.  SMART says
> there's no problems with the drives.
> 
> A resync/recover operation typically proceeds at an average
> speed of about 35MB/sec, as reported by /proc/mdstat.  But
> then - for the times that it hung - /proc/mdstat reports slower
> and slower speeds and longer and longer finish times (30,000 minutes
> plus!).  In /sys/block/md1/md the value of sync_completed
> would stay static and sync_speed would drop lower and lower
> (< 1000KB/sec).
> 
> I tried:
> 
>  echo 40960 > sync_speed_min
> 
> in an attempt to try and coax things to go faster but the system
> remained hung.

If the hardware setup cannot do faster than 35 MB/second nothing you can do will 
make it go faster.

> 
> The system was hung in that:
> 
> o  load average increased to about 13; top reported 50% spent
> in 'wait time';
> 
> o  Any operation that accessed the disk/md device would 'hang'.
> Other trivial operations - shell builtin commands, X11 widget
> updates - still worked.  'shutdown -r now' wouldn't work; I had
> to cold-boot the system each time.
> 
> o  No error messages logged to the console or syslog.
> 
> This 'hang' *seems* to be related to system activity; the system
> has never been *heavily* loaded the three times a resync/recover
> operation failed but I had a couple of download programs and the
> like - keeping the network interface mildly busy - running in every
> failed/hung case.
> 
> Ideally the resync/recover operation should proceed independent
> of the system activity, I would have thought?  I'd hoped to be able to
> perform daily/weekly transparent backups by plugging in the third drive,
> adding it to the raid1 md device and then detaching the disk
> after the recover operation had completed.
> 
> Can anyone help?  I have no idea if there are other things I can
> do or tune to get around this problem, or if it's an actual bug.  I had
> a look in the kernel archives but couldn't see anything that seemed
> relevant to this problem with the latest stable kernel.
> 
> Thanks,

It really sounds like a HW issue of some sort, a weak power supply, or a badly 
designed MB.  I have a badly designed one here that if I put disks on certain 
built-in MB ports then things work just fine for weeks and weeks until something 
happens to cause a resync, and on any resync the MTBF is <30 minutes, moving the 
exact same disks off of those built-in MB ports makes things work just fine, and 
having the disk on the built-in MB ports also results in other cards in the PCI 
bus having serious issues (losing data and other crap).

What kind of motherboard is it and which chipset is on that MB, and what kind of 
sata ports are the disks on?

If you can get all 3 disks in the machine and working (without a resync) then I 
would try doing "dd if=/dev/sda of=/dev/null bs=1M" for each of the 3 disks at 
the same time and see if that causes a hang, if it does it is not a MD issue, 
also I would check the speed of 1 disk, 2 disks, and 3 disks and see how bad 
those ports bottleneck with multiple disks being used.

                         Roger

Re: System hangs on raid md recovery/resync

[Brad]

Roger, thanks for your reply.

> If the hardware setup cannot do faster than 35 MB/second nothing you can do
> will make it go faster.

On a straight read of any of the disks, using dd or 'hdparm -t', I can get about
70MB/sec to 75MB/sec out of the disk drives.  I've always thought that the
MD software deliberately 'throttled' a resync/recover operation, slowing things
down if the disks were being used by applications, so I've been happy
with around
35MB/sec for my resyncs, as the system has usually been lightly loaded
with other
programs using the drives.

> What kind of motherboard is it and which chipset is on that MB, and what
> kind of sata ports are the disks on?
>
> If you can get all 3 disks in the machine and working (without a resync)
> then I would try doing "dd if=/dev/sda of=/dev/null bs=1M" for each of the 3
> disks at the same time and see if that causes a hang, if it does it is not a
> MD issue, also I would check the speed of 1 disk, 2 disks, and 3 disks and
> see how bad those ports bottleneck with multiple disks being used.

My system is 7 months old; the motherboard is a Gigabyte GA-P35-DS4.
It has an Intel ICH9R northbridge with 6 SATA 2 ports and a 'Gigabyte'
(JMicron 20360/20363) southbridge with 2 SATA 2 ports.  I have two
500GB Western Digital SATA 2 internal disks, one on each controller, in
an MD raid1 mirror.  I've experienced these problems while plugging in
a third Western Digital 500GB drive into the ICH9R controller and
adding it as a third mirror element to the raid1 MD device.

Other than this 'hang' problem with MD I've never had a problem with
any of the disks.  For example, after yesterday failing to synchronise the third
disk with the MD raid1 device, I proceeded to do a filesystem-level copy, using
cpio to copy all the files from the MD device to the (separately mounted) third
disk.  That worked fine (took a lot longer, though, because of the huge number
of small files I have on the filesystem).  A follow-up rsync to
'catch' files that
had been modified during the cpio also succeeded.

I just now ran a dd test as you suggested of each disk, and each ran fine, with
dd reporting speeds of 66.1, 68.6 and 70.5 MB/s.  One 'hard resetting link'
error/event was logged for one of the three SATA 2 ports without the dd process
for that link seeing any error.  I saw absolutely no such errors
logged at all with
my 'hang' problems in synchronising the raid1 device yesterday.  Everything
would proceed fine until the resync operation simply stopped - with
/sys/block/md1/md/sync_completed static, showing no further
progress - and the system then 'hanging' on anything that tried
to access a disk.


Brad

Re: System hangs on raid md recovery/resync

[Roger Heflin]

Brad wrote:
> Roger, thanks for your reply.

> 
> My system is 7 months old; the motherboard is a Gigabyte GA-P35-DS4.
> It has an Intel ICH9R northbridge with 6 SATA 2 ports and a 'Gigabyte'
> (JMicron 20360/20363) southbridge with 2 SATA 2 ports.  I have two
> 500GB Western Digital SATA 2 internal disks, one on each controller, in
> an MD raid1 mirror.  I've experienced these problems while plugging in
> a third Western Digital 500GB drive into the ICH9R controller and
> adding it as a third mirror element to the raid1 MD device.
> 
> Other than this 'hang' problem with MD I've never had a problem with
> any of the disks.  For example, after yesterday failing to synchronise the third
> disk with the MD raid1 device, I proceeded to do a filesystem-level copy, using
> cpio to copy all the files from the MD device to the (separately mounted) third
> disk.  That worked fine (took a lot longer, though, because of the huge number
> of small files I have on the filesystem).  A follow-up rsync to
> 'catch' files that
> had been modified during the cpio also succeeded.
> 
> I just now ran a dd test as you suggested of each disk, and each ran fine, with
> dd reporting speeds of 66.1, 68.6 and 70.5 MB/s.  One 'hard resetting link'
> error/event was logged for one of the three SATA 2 ports without the dd process
> for that link seeing any error.  I saw absolutely no such errors
> logged at all with
> my 'hang' problems in synchronising the raid1 device yesterday.  Everything
> would proceed fine until the resync operation simply stopped - with
> /sys/block/md1/md/sync_completed static, showing no further
> progress - and the system then 'hanging' on anything that tried
> to access a disk.


The Intel stuff tends to be pretty decent, where I have ran into the most issues 
is with anything that the MB vendor adds on, so I would try putting all 3 on the 
Intel, and in the past the Intel controllers I have tested have been able to run 
all disks at full speed (or close to it) even when multiple disks are being 
actively used, this would at least eliminate the jmicron controller from the mix.

The sync will slow down quite a bit if other things are causing reads/writes 
that are causing the disks to seek around.

How much power does the PS have on the 12V line?    So long as it is either a 
split 12V supply or has more than 15-20A (non-split PS) you should be OK.  I 
have had issues with non-split PS's that only had 15A on the 12V resulting in 
odd happenings with the disk.

I would also setup the sysrq keys and the next time it happens do a "alt-sysrq" 
with a "T" "W" and a "Q" to test sysrq and make sure it is working do a 
"alt-sysrq-S" (sync) as this should produces messages in dmesg and show that 
things are enabled and working.

You did run the dd on all 3 disks at the same time?

The hard resetting link usually indicates something bad happened, though that 
could be caused by a lot of things.

                                 Roger

Re: System hangs on raid md recovery/resync

[Brad]

Roger,

> The Intel stuff tends to be pretty decent, where I have ran into the most
> issues is with anything that the MB vendor adds on, so I would try putting
> all 3 on the Intel, and in the past the Intel controllers I have tested have
> been able to run all disks at full speed (or close to it) even when multiple
> disks are being actively used, this would at least eliminate the jmicron
> controller from the mix.

I thought I was being cute in putting the two disks of the regular raid1
set on different controllers, for maximum redundancy.  :-)  But yes, the
JMicron controller seems to be a bit flakey ... sometimes after a reboot it will
experience constant 'hard resetting link' errors under full load.  After another
reboot though it'll be as stable as a rock.  I've assumed Linux isn't getting
something quite right when it initialises the JMicron controller at boot.

In any case I've noticed - using iostat - that when I've added the third drive
to the raid1 device, for example, the recover operation consists of reads on
the disk that's hooked up to the ICH9R controller and writes to the third
disk on the same controller.  The MD code doesn't seem to share the read
operation between the two existing mirrored disks, so the second disk on
the JMicron isn't involved at all.

> How much power does the PS have on the 12V line?    So long as it is either
> a split 12V supply or has more than 15-20A (non-split PS) you should be OK.

I'm sorry but I wouldn't have a clue about that.  I've got an Antec
Sonata III 500
case with its standard 500W power suppler.  "80 PLUS" certified, whatever
that means ... "an EarthWatts 500 Watt power supply unit (PSU) which is
equipped with universal input and Active PFC. This PSU is also 80PLUS(R)
certified making it one of the most efficient PSU's available" says their web
site.

> You did run the dd on all 3 disks at the same time?

Yes.

> The hard resetting link usually indicates something bad happened, though
> that could be caused by a lot of things.

I normally never see that error at all.  And I didn't see it the three times the
system hung on the MD raid1 resync/recover operations.

If the kernel MD code had a hardware problem with the SATA 2 ports would
I see an error message?


Brad

Re: System hangs on raid md recovery/resync

[Roger Heflin]

Brad wrote:
> Roger,
> 
>> The Intel stuff tends to be pretty decent, where I have ran into the most
>> issues is with anything that the MB vendor adds on, so I would try putting
>> all 3 on the Intel, and in the past the Intel controllers I have tested have
>> been able to run all disks at full speed (or close to it) even when multiple
>> disks are being actively used, this would at least eliminate the jmicron
>> controller from the mix.
> 
> I thought I was being cute in putting the two disks of the regular raid1
> set on different controllers, for maximum redundancy.  :-)  But yes, the
> JMicron controller seems to be a bit flakey ... sometimes after a reboot it will
> experience constant 'hard resetting link' errors under full load.  After another
> reboot though it'll be as stable as a rock.  I've assumed Linux isn't getting
> something quite right when it initialises the JMicron controller at boot.
> 
> In any case I've noticed - using iostat - that when I've added the third drive
> to the raid1 device, for example, the recover operation consists of reads on
> the disk that's hooked up to the ICH9R controller and writes to the third
> disk on the same controller.  The MD code doesn't seem to share the read
> operation between the two existing mirrored disks, so the second disk on
> the JMicron isn't involved at all.

 From a performance stand point and if everything works fine it is a good idea, 
from a debugging and potential bug stand point, more drivers/more separate 
hardware involves more code/hardware so there is more stuff to have a bug/design 
defect.  With everything being on the Intel sata controllers I would not expect 
it to make a performance difference either way.

It could be the other traffic hitting the jmicron causes the failure when the 
Intel one is busy.

On mine the issues seems to be that the MB system does not properly deal with 
things being heavily utilized (and I have seen at least 2-3 other MB's that as 
designed only fail when pushed close to the bandwidth limit of the internal 
buses), if things are not pushed too hard it will run fine for weeks with no 
errors, push it hard enough the right way (and in my case a raid5 MD rebuild is 
what does it) and you can make it fall over in just minutes with any other 
traffic outside the MD subsystem also needing to be there.

> 
>> How much power does the PS have on the 12V line?    So long as it is either
>> a split 12V supply or has more than 15-20A (non-split PS) you should be OK.
> 
> I'm sorry but I wouldn't have a clue about that.  I've got an Antec
> Sonata III 500
> case with its standard 500W power suppler.  "80 PLUS" certified, whatever
> that means ... "an EarthWatts 500 Watt power supply unit (PSU) which is
> equipped with universal input and Active PFC. This PSU is also 80PLUS(R)
> certified making it one of the most efficient PSU's available" says their web
> site.

I have a earthwatts 380, the 500 should be just fine, that is what is called a 
split rail supply, it has 2 separate 12V supplies.    The ratings would be on 
the PS box itself, and on the PS itself.    My 380 says 12V(v1) 17A and 12V(v2) 
17A, a non-split rail supply will have only one 12V entry and it may be under 
17A on the single supply.

> 
>> You did run the dd on all 3 disks at the same time?
> 
> Yes.
> 
>> The hard resetting link usually indicates something bad happened, though
>> that could be caused by a lot of things.
> 
> I normally never see that error at all.  And I didn't see it the three times the
> system hung on the MD raid1 resync/recover operations.
> 
> If the kernel MD code had a hardware problem with the SATA 2 ports would
> I see an error message?

The message would go on the text console, if it hangs the system bad enough it 
would likely never make it to the log files as the disk subsystem is hung, so it 
would not be in the log files.   Also if you can switch to the console after the 
event the message would not be there, you would need to already have the text 
console up when the error happens.   dmesg may have the message if enough of the 
system is still working for dmesg to run.

                                 Roger