Query regarding disk failure

Query regarding disk failure

[Priya Kamala]

Hi,

If a single disk develops bad sector(s) that affects only a single 
stripe within a given RAID-5 configuration and these bad sectors are 
discovered when running a "check/resync" (echo check > 
/sys/block/mdX/md/sync_action), does this result in the entire disk 
being removed from the RAID? In other words, is access to the other good 
stripes on the drive disabled as well? If so, why is it done this way? 
In case the kernel version makes a difference to the behaviour, I'm 
working with 2.6.21.1.

Thanks,
Priya

Re: Query regarding disk failure

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 1560 bytes --]

On Wed Jul 10, 2013 at 01:09:20PM +0100, Priya Kamala wrote:

> Hi,
> 
> If a single disk develops bad sector(s) that affects only a single 
> stripe within a given RAID-5 configuration and these bad sectors are 
> discovered when running a "check/resync" (echo check > 
> /sys/block/mdX/md/sync_action), does this result in the entire disk 
> being removed from the RAID? In other words, is access to the other good 
> stripes on the drive disabled as well? If so, why is it done this way? 
> In case the kernel version makes a difference to the behaviour, I'm 
> working with 2.6.21.1.
> 
If the check/repair process finds a read error, it will trigger a
rewrite of the affected block (rebuilding the data from the other
drives). If the write fails then the disk will be removed from the RAID.
This shouldn't happen, as the disk should either succeed in the write,
or should remap the bad block with one of its spares.

If you're getting repeated problems with this, it's likely that you are
using consumer-level drives without TLER/ERC and have failed to adjust
the kernel timeouts to compensate for this. In this case the rewrite
will be triggered while the disk is still attempting the read process
and will therefore be ignored, causing the write to fail and the disk to
be removed.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Query regarding disk failure

[Priya Kamala]

On 10/07/2013 14:11, Robin Hill wrote:
> If the check/repair process finds a read error, it will trigger a
> rewrite of the affected block (rebuilding the data from the other
> drives). If the write fails then the disk will be removed from the RAID.
> This shouldn't happen, as the disk should either succeed in the write,
> or should remap the bad block with one of its spares.
>
> If you're getting repeated problems with this, it's likely that you are
> using consumer-level drives without TLER/ERC and have failed to adjust
> the kernel timeouts to compensate for this. In this case the rewrite
> will be triggered while the disk is still attempting the read process
> and will therefore be ignored, causing the write to fail and the disk to
> be removed.
>
Thank you for the quick response. The drives in question do support ERC.

I have some additional questions. Is the drive status changed to an 
underscore in /proc/mdstat during this remap process?

This is an extract from the system log:

Apr  4 07:44:02 src@AA20NOV sd 8:0:0:0: SCSI error: return code = 0x08000002
Apr  4 07:44:02 src@AA20NOV sde: Current: sense key: Medium Error
Apr  4 07:44:02 src@AA20NOV Additional sense: No additional sense 
information
Apr  4 07:44:02 src@AA20NOV Info fld=0x620cb71
Apr  4 07:44:02 src@AA20NOV end_request: I/O error, dev sde, sector 
102812529
Apr  4 07:44:02 src@AA20NOV raid5:md3: read error corrected (8 sectors 
at 86035184 on sde5)
Apr  4 07:44:02 src@AA20NOV raid5:md3: read error corrected (8 sectors 
at 86035192 on sde5)
Apr  4 07:44:05 src@AA20NOV raid5: Disk failure on sde5, disabling 
device. Operation continuing on 4 devices
Apr  4 07:44:05 src@AA20NOV md: cannot remove active disk sde5 from md3 ...

We have a service which monitors the md RAID status and tries to remove 
failed disks from degraded RAIDs and it looks like that failed since the 
device is active. But why would it still be active when the previous 
line says that the device is being disabled?

Thanks,
Priya

Re: Query regarding disk failure

[Phil Turmel]

On 07/10/2013 10:17 AM, Priya Kamala wrote:
> 
> On 10/07/2013 14:11, Robin Hill wrote:
>> If the check/repair process finds a read error, it will trigger a
>> rewrite of the affected block (rebuilding the data from the other
>> drives). If the write fails then the disk will be removed from the RAID.
>> This shouldn't happen, as the disk should either succeed in the write,
>> or should remap the bad block with one of its spares.
>>
>> If you're getting repeated problems with this, it's likely that you are
>> using consumer-level drives without TLER/ERC and have failed to adjust
>> the kernel timeouts to compensate for this. In this case the rewrite
>> will be triggered while the disk is still attempting the read process
>> and will therefore be ignored, causing the write to fail and the disk to
>> be removed.
>>
> Thank you for the quick response. The drives in question do support ERC.

You haven't actually told us whether it is turned on.  Desktop drives
that support it don't have it enabled by default.  Please show "smartctl
-l scterc /dev/sdX" for each of your drives.

Also, while read errors will normally be rewritten, there is a kernel
limit of 20 read errors in an hour, and 10 per hour after that.  So if
your check operation is encountering numerous UREs, it might be hitting
this limit.

> I have some additional questions. Is the drive status changed to an
> underscore in /proc/mdstat during this remap process?

No.

> This is an extract from the system log:
> 
> Apr  4 07:44:02 src@AA20NOV sd 8:0:0:0: SCSI error: return code =
> 0x08000002
> Apr  4 07:44:02 src@AA20NOV sde: Current: sense key: Medium Error
> Apr  4 07:44:02 src@AA20NOV Additional sense: No additional sense
> information
> Apr  4 07:44:02 src@AA20NOV Info fld=0x620cb71
> Apr  4 07:44:02 src@AA20NOV end_request: I/O error, dev sde, sector
> 102812529
> Apr  4 07:44:02 src@AA20NOV raid5:md3: read error corrected (8 sectors
> at 86035184 on sde5)
> Apr  4 07:44:02 src@AA20NOV raid5:md3: read error corrected (8 sectors
> at 86035192 on sde5)
> Apr  4 07:44:05 src@AA20NOV raid5: Disk failure on sde5, disabling
> device. Operation continuing on 4 devices
> Apr  4 07:44:05 src@AA20NOV md: cannot remove active disk sde5 from md3 ...

You haven't really shown enough, but this looks like the tail of the
event sequence when ERC is disabled or unavailable.

Phil

Re: Query regarding disk failure

[Priya Kamala]

On 10/07/2013 15:51, Phil Turmel wrote:
> You haven't actually told us whether it is turned on.  Desktop drives
> that support it don't have it enabled by default.  Please show "smartctl
> -l scterc /dev/sdX" for each of your drives.
I don't have access to the original system right now. This is the output 
from the another system with the same drive type. All the drives show 
the same output, so I'm including only one. The drives are Seagate 
Barracuda ES.2 Model ST31000340NS.

-bash-4.1# smartctl -l scterc /dev/sda
smartctl 5.43 2012-06-30 r3573 [i686-linux-3.2.40] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

SCT Error Recovery Control:
            Read:    100 (10.0 seconds)
           Write:    100 (10.0 seconds)

Thanks,
Priya

Re: Query regarding disk failure

[Phil Turmel]

On 07/10/2013 11:11 AM, Priya Kamala wrote:
> 
> On 10/07/2013 15:51, Phil Turmel wrote:
>> You haven't actually told us whether it is turned on.  Desktop drives
>> that support it don't have it enabled by default.  Please show "smartctl
>> -l scterc /dev/sdX" for each of your drives.
> I don't have access to the original system right now. This is the output
> from the another system with the same drive type. All the drives show
> the same output, so I'm including only one. The drives are Seagate
> Barracuda ES.2 Model ST31000340NS.

Ok, enterprise drives.  ERC is on by default.

> -bash-4.1# smartctl -l scterc /dev/sda
> smartctl 5.43 2012-06-30 r3573 [i686-linux-3.2.40] (local build)
> Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
> 
> SCT Error Recovery Control:
>            Read:    100 (10.0 seconds)
>           Write:    100 (10.0 seconds)

Did you have an idea of number of read error events?  "Pending" sector
counts on the problem drive might be a useful number to report.  (Or
just report complete output of "smartctl -x" for each drive.)

Meanwhile, what distro, kernel, and mdadm versions are involved here?

Phil

Re: Query regarding disk failure

[Priya Kamala]

On 10/07/2013 16:22, Phil Turmel wrote:
> Did you have an idea of number of read error events?  "Pending" sector
> counts on the problem drive might be a useful number to report.  (Or
> just report complete output of "smartctl -x" for each drive.)
Unfortunately I don't have this information, as this is a post-mortem 
analysis of the failure on a customer system.
> Meanwhile, what distro, kernel, and mdadm versions are involved here?
I'm working with a distro based on Centos running 2.6.21.1 kernel and 
mdadm 2.6.

Thanks,
Priya

Re: Query regarding disk failure

[Phil Turmel]

On 07/10/2013 11:45 AM, Priya Kamala wrote:
> On 10/07/2013 16:22, Phil Turmel wrote:
>> Did you have an idea of number of read error events?  "Pending" sector
>> counts on the problem drive might be a useful number to report.  (Or
>> just report complete output of "smartctl -x" for each drive.)
> Unfortunately I don't have this information, as this is a post-mortem
> analysis of the failure on a customer system.
>> Meanwhile, what distro, kernel, and mdadm versions are involved here?
> I'm working with a distro based on Centos running 2.6.21.1 kernel and
> mdadm 2.6.

That explains your experience.  Your kernel dates from April 2007.
Tolerance for corrected read errors was added in December 2009.  (Robert
Becker, 1e50915f, to be specific.)

You need a newer kernel.

Phil

Re: Query regarding disk failure

[Mikael Abrahamsson]

On Wed, 10 Jul 2013, Priya Kamala wrote:

> If a single disk develops bad sector(s) that affects only a single 
> stripe within a given RAID-5 configuration and these bad sectors are 
> discovered when running a "check/resync" (echo check > 
> /sys/block/mdX/md/sync_action), does this result in the entire disk 
> being removed from the RAID? In other words, is access to the other good 
> stripes on the drive disabled as well? If so, why is it done this way? 
> In case the kernel version makes a difference to the behaviour, I'm 
> working with 2.6.21.1.

If drive is kicked, it could be because it's a consumer drive which 
doesn't support TLER/ERC.

Do:

for x in /sys/block/sd[a-z] ; do echo 180  > $x/device/timeout ; done

This should mean the disk isn't kicked from the array unless it's really 
broken.

But as was mentioned in another mail in your thread, you really should 
upgrade your kernel. It's really old.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se