devices get kicked from RAID about once a month

devices get kicked from RAID about once a month

[Dan Christensen]

Over the past 5 months, I've had a drive booted from one of my raid
arrays about 6 times.  In each case, the drive passes SMART tests, so I
--remove it, --re-add it, and it resyncs successfully.

I tried disconnecting and re-connecting all four SATA cables, but the
problem occurred again.  In fact, today *two* partitions were kicked out
of their (different) raid devices.

All of the problems occurred with sda and sdc, which are older drives:

sda:     SAMSUNG SP2004C
sdc:     SAMSUNG SP2504C

hddtemp shows the temperatures at 32C.

System runs Debian lenny, with newer kernel than lenny: 2.6.28.
mdadm version v2.6.7.2.

Motherboard is a Gigabyte GA-E7AUM-DS2H.  I couldn't find the controller
chipset info.

Are the drives just bad?  Or is it the controller?

More detailed information is below.  Thanks for any help!  Let me know
if I should provide more information.

Dan

syslog messages from today:

Jun  2 03:54:22 boots kernel: [66986.000043] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Jun  2 03:54:23 boots kernel: [66986.000052] ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Jun  2 03:54:23 boots kernel: [66986.000053]          res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Jun  2 03:54:23 boots kernel: [66986.000056] ata1.00: status: { DRDY }
Jun  2 03:54:23 boots kernel: [66986.000064] ata1: hard resetting link
Jun  2 03:54:23 boots kernel: [66986.484037] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Jun  2 03:54:23 boots kernel: [66986.494003] ata1.00: configured for UDMA/133
Jun  2 03:54:23 boots kernel: [66986.494016] end_request: I/O error, dev sda, sector 187880006
Jun  2 03:54:23 boots kernel: [66986.494023] md: super_written gets error=-5, uptodate=0
Jun  2 03:54:23 boots kernel: [66986.494027] raid5: Disk failure on sda7, disabling device.
Jun  2 03:54:24 boots kernel: [66986.494029] raid5: Operation continuing on 3 devices.
Jun  2 03:54:24 boots kernel: [66986.494045] ata1: EH complete
Jun  2 03:54:24 boots kernel: [66986.494215] sd 0:0:0:0: [sda] 390719855 512-byte hardware sectors: (200 GB/186 GiB)
Jun  2 03:54:24 boots kernel: [66986.494244] sd 0:0:0:0: [sda] Write Protect is off
Jun  2 03:54:24 boots kernel: [66986.494248] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
Jun  2 03:54:24 boots kernel: [66986.494274] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun  2 03:54:24 boots kernel: [66986.762936] RAID5 conf printout:
Jun  2 03:54:24 boots mdadm[4109]: Fail event detected on md device /dev/md3, component device /dev/sda7
Jun  2 03:54:24 boots kernel: [66986.762942]  --- rd:4 wd:3
Jun  2 03:54:24 boots kernel: [66986.762946]  disk 0, o:0, dev:sda7
Jun  2 03:54:24 boots kernel: [66986.762948]  disk 1, o:1, dev:sdb3
Jun  2 03:54:24 boots kernel: [66986.762950]  disk 2, o:1, dev:sdc5
Jun  2 03:54:24 boots kernel: [66986.762953]  disk 3, o:1, dev:sdd3
Jun  2 03:54:24 boots kernel: [66986.763626] RAID5 conf printout:
Jun  2 03:54:24 boots kernel: [66986.763628]  --- rd:4 wd:3
Jun  2 03:54:24 boots kernel: [66986.763630]  disk 1, o:1, dev:sdb3
Jun  2 03:54:24 boots kernel: [66986.763632]  disk 2, o:1, dev:sdc5
Jun  2 03:54:24 boots kernel: [66986.763634]  disk 3, o:1, dev:sdd3

Jun  2 06:59:33 boots kernel: [78097.000087] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Jun  2 06:59:34 boots kernel: [78097.000095] ata4.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Jun  2 06:59:34 boots kernel: [78097.000096]          res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Jun  2 06:59:34 boots kernel: [78097.000099] ata4.00: status: { DRDY }
Jun  2 06:59:34 boots kernel: [78097.000106] ata4: hard resetting link
Jun  2 06:59:34 boots kernel: [78097.484057] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Jun  2 06:59:35 boots kernel: [78097.493930] ata4.00: configured for UDMA/133
Jun  2 06:59:35 boots kernel: [78097.493941] end_request: I/O error, dev sdc, sector 488391944
Jun  2 06:59:35 boots kernel: [78097.493947] md: super_written gets error=-5, uptodate=0
Jun  2 06:59:35 boots kernel: [78097.493952] raid5: Disk failure on sdc7, disabling device.
Jun  2 06:59:35 boots kernel: [78097.493953] raid5: Operation continuing on 2 devices.
Jun  2 06:59:35 boots kernel: [78097.493967] ata4: EH complete
Jun  2 06:59:35 boots kernel: [78097.494105] sd 3:0:0:0: [sdc] 488397168 512-byte hardware sectors: (250 GB/232 GiB)
Jun  2 06:59:35 boots kernel: [78097.494124] sd 3:0:0:0: [sdc] Write Protect is off
Jun  2 06:59:35 boots kernel: [78097.494127] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
Jun  2 06:59:35 boots kernel: [78097.494156] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun  2 06:59:35 boots mdadm[4109]: Fail event detected on md device /dev/md5, component device /dev/sdc7
Jun  2 06:59:35 boots kernel: [78097.635934] RAID5 conf printout:
Jun  2 06:59:35 boots kernel: [78097.635938]  --- rd:3 wd:2
Jun  2 06:59:35 boots kernel: [78097.635941]  disk 0, o:1, dev:sdb6
Jun  2 06:59:35 boots kernel: [78097.635944]  disk 1, o:0, dev:sdc7
Jun  2 06:59:35 boots kernel: [78097.635946]  disk 2, o:1, dev:sdd6
Jun  2 06:59:36 boots kernel: [78097.636143] RAID5 conf printout:
Jun  2 06:59:36 boots kernel: [78097.636146]  --- rd:3 wd:2
Jun  2 06:59:36 boots kernel: [78097.636148]  disk 0, o:1, dev:sdb6
Jun  2 06:59:36 boots kernel: [78097.636150]  disk 2, o:1, dev:sdd6

------------------------

/proc/mdstat:

Personalities : [raid1] [raid6] [raid5] [raid4] 
md6 : active raid1 sdb7[0] sdd7[1]
      196290048 blocks [2/2] [UU]
      bitmap: 1/3 pages [4KB], 32768KB chunk

md5 : active raid5 sdc7[3] sdb6[0] sdd6[2]
      175815168 blocks level 5, 64k chunk, algorithm 2 [3/2] [U_U]
      [=================>...]  recovery = 89.3% (78552568/87907584) finish=4.6min speed=33323K/sec
      bitmap: 1/2 pages [4KB], 32768KB chunk

md4 : active raid5 sda8[0] sdd5[3] sdc6[2] sdb5[1]
      218636160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/2 pages [0KB], 32768KB chunk

md3 : active raid5 sda7[4] sdd3[3] sdc5[2] sdb3[1]
      218612160 blocks level 5, 64k chunk, algorithm 2 [4/3] [_UUU]
        resync=DELAYED
      bitmap: 2/2 pages [8KB], 32768KB chunk

md2 : active raid5 sda6[0] sdd2[3] sdc2[2] sdb2[1]
      30748032 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 1/1 pages [4KB], 32768KB chunk

md0 : active raid5 sda2[0] sdd1[2] sdc1[1]
      578048 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      bitmap: 0/1 pages [0KB], 32768KB chunk

md1 : active raid1 sdb1[0] sda5[1]
      289024 blocks [2/2] [UU]
      bitmap: 0/1 pages [0KB], 32768KB chunk

unused devices: <none>

------------------

/etc/mdadm/mdadm.conf:

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR jdc@uwo.ca

# definitions of existing MD arrays
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=6b8b4567:327b23c6:643c9869:66334873
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=ba493129:00074cd3:fee07e15:038135d5
ARRAY /dev/md2 level=raid5 num-devices=4 UUID=3dc9b50b:b9270472:9778d943:b967813b
ARRAY /dev/md3 level=raid5 num-devices=4 UUID=c4056d19:7b4bb550:44925b88:91d5bc8a
ARRAY /dev/md4 level=raid5 num-devices=4 UUID=d7c84402:210b78c7:556bbbc0:47df436c
ARRAY /dev/md5 level=raid5 num-devices=3 UUID=9effd43f:93ccc32d:899ca6c7:ea966964
ARRAY /dev/md6 level=raid1 num-devices=2 UUID=da17264f:be7e012d:85187211:fb0e2ebd

Re: devices get kicked from RAID about once a month

[rsivak]

I would say that these are probably desktop type drives - not enterprise drives.  In which case, what you're probably hitting are media errors - when a drive hits a bad sector.  A consumer level drive will try as hard as possible to retrieve the data - I'd say somewhere around 15 seconds, depending on the manufacturer.  This is often enough to get the drive kicked out of the array.  

Enterprise level drives usually ship with this feature disabled, so that the raid can take care of it.  

Some manufacturers provide utilities to manage this feature.  Until late last year, for example, western digital had a utility called WDTLER.  According to user reports, however, the utility no longer works on wd drives manufactured after Nov 2009.

I suggest you see if your manufacturer provides a similar utility.

Russ
Sent from my Verizon Wireless BlackBerry

-----Original Message-----
From: Dan Christensen <jdc@uwo.ca>
Date:	Wed, 02 Jun 2010 10:14:28 
To: <linux-raid@vger.kernel.org>
Subject: devices get kicked from RAID about once a month

Over the past 5 months, I've had a drive booted from one of my raid
arrays about 6 times.  In each case, the drive passes SMART tests, so I
--remove it, --re-add it, and it resyncs successfully.

I tried disconnecting and re-connecting all four SATA cables, but the
problem occurred again.  In fact, today *two* partitions were kicked out
of their (different) raid devices.

All of the problems occurred with sda and sdc, which are older drives:

sda:     SAMSUNG SP2004C
sdc:     SAMSUNG SP2504C

hddtemp shows the temperatures at 32C.

System runs Debian lenny, with newer kernel than lenny: 2.6.28.
mdadm version v2.6.7.2.

Motherboard is a Gigabyte GA-E7AUM-DS2H.  I couldn't find the controller
chipset info.

Are the drives just bad?  Or is it the controller?

More detailed information is below.  Thanks for any help!  Let me know
if I should provide more information.

Dan

syslog messages from today:

Jun  2 03:54:22 boots kernel: [66986.000043] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Jun  2 03:54:23 boots kernel: [66986.000052] ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Jun  2 03:54:23 boots kernel: [66986.000053]          res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Jun  2 03:54:23 boots kernel: [66986.000056] ata1.00: status: { DRDY }
Jun  2 03:54:23 boots kernel: [66986.000064] ata1: hard resetting link
Jun  2 03:54:23 boots kernel: [66986.484037] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Jun  2 03:54:23 boots kernel: [66986.494003] ata1.00: configured for UDMA/133
Jun  2 03:54:23 boots kernel: [66986.494016] end_request: I/O error, dev sda, sector 187880006
Jun  2 03:54:23 boots kernel: [66986.494023] md: super_written gets error=-5, uptodate=0
Jun  2 03:54:23 boots kernel: [66986.494027] raid5: Disk failure on sda7, disabling device.
Jun  2 03:54:24 boots kernel: [66986.494029] raid5: Operation continuing on 3 devices.
Jun  2 03:54:24 boots kernel: [66986.494045] ata1: EH complete
Jun  2 03:54:24 boots kernel: [66986.494215] sd 0:0:0:0: [sda] 390719855 512-byte hardware sectors: (200 GB/186 GiB)
Jun  2 03:54:24 boots kernel: [66986.494244] sd 0:0:0:0: [sda] Write Protect is off
Jun  2 03:54:24 boots kernel: [66986.494248] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
Jun  2 03:54:24 boots kernel: [66986.494274] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun  2 03:54:24 boots kernel: [66986.762936] RAID5 conf printout:
Jun  2 03:54:24 boots mdadm[4109]: Fail event detected on md device /dev/md3, component device /dev/sda7
Jun  2 03:54:24 boots kernel: [66986.762942]  --- rd:4 wd:3
Jun  2 03:54:24 boots kernel: [66986.762946]  disk 0, o:0, dev:sda7
Jun  2 03:54:24 boots kernel: [66986.762948]  disk 1, o:1, dev:sdb3
Jun  2 03:54:24 boots kernel: [66986.762950]  disk 2, o:1, dev:sdc5
Jun  2 03:54:24 boots kernel: [66986.762953]  disk 3, o:1, dev:sdd3
Jun  2 03:54:24 boots kernel: [66986.763626] RAID5 conf printout:
Jun  2 03:54:24 boots kernel: [66986.763628]  --- rd:4 wd:3
Jun  2 03:54:24 boots kernel: [66986.763630]  disk 1, o:1, dev:sdb3
Jun  2 03:54:24 boots kernel: [66986.763632]  disk 2, o:1, dev:sdc5
Jun  2 03:54:24 boots kernel: [66986.763634]  disk 3, o:1, dev:sdd3

Jun  2 06:59:33 boots kernel: [78097.000087] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Jun  2 06:59:34 boots kernel: [78097.000095] ata4.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Jun  2 06:59:34 boots kernel: [78097.000096]          res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Jun  2 06:59:34 boots kernel: [78097.000099] ata4.00: status: { DRDY }
Jun  2 06:59:34 boots kernel: [78097.000106] ata4: hard resetting link
Jun  2 06:59:34 boots kernel: [78097.484057] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Jun  2 06:59:35 boots kernel: [78097.493930] ata4.00: configured for UDMA/133
Jun  2 06:59:35 boots kernel: [78097.493941] end_request: I/O error, dev sdc, sector 488391944
Jun  2 06:59:35 boots kernel: [78097.493947] md: super_written gets error=-5, uptodate=0
Jun  2 06:59:35 boots kernel: [78097.493952] raid5: Disk failure on sdc7, disabling device.
Jun  2 06:59:35 boots kernel: [78097.493953] raid5: Operation continuing on 2 devices.
Jun  2 06:59:35 boots kernel: [78097.493967] ata4: EH complete
Jun  2 06:59:35 boots kernel: [78097.494105] sd 3:0:0:0: [sdc] 488397168 512-byte hardware sectors: (250 GB/232 GiB)
Jun  2 06:59:35 boots kernel: [78097.494124] sd 3:0:0:0: [sdc] Write Protect is off
Jun  2 06:59:35 boots kernel: [78097.494127] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
Jun  2 06:59:35 boots kernel: [78097.494156] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun  2 06:59:35 boots mdadm[4109]: Fail event detected on md device /dev/md5, component device /dev/sdc7
Jun  2 06:59:35 boots kernel: [78097.635934] RAID5 conf printout:
Jun  2 06:59:35 boots kernel: [78097.635938]  --- rd:3 wd:2
Jun  2 06:59:35 boots kernel: [78097.635941]  disk 0, o:1, dev:sdb6
Jun  2 06:59:35 boots kernel: [78097.635944]  disk 1, o:0, dev:sdc7
Jun  2 06:59:35 boots kernel: [78097.635946]  disk 2, o:1, dev:sdd6
Jun  2 06:59:36 boots kernel: [78097.636143] RAID5 conf printout:
Jun  2 06:59:36 boots kernel: [78097.636146]  --- rd:3 wd:2
Jun  2 06:59:36 boots kernel: [78097.636148]  disk 0, o:1, dev:sdb6
Jun  2 06:59:36 boots kernel: [78097.636150]  disk 2, o:1, dev:sdd6

------------------------

/proc/mdstat:

Personalities : [raid1] [raid6] [raid5] [raid4] 
md6 : active raid1 sdb7[0] sdd7[1]
      196290048 blocks [2/2] [UU]
      bitmap: 1/3 pages [4KB], 32768KB chunk

md5 : active raid5 sdc7[3] sdb6[0] sdd6[2]
      175815168 blocks level 5, 64k chunk, algorithm 2 [3/2] [U_U]
      [=================>...]  recovery = 89.3% (78552568/87907584) finish=4.6min speed=33323K/sec
      bitmap: 1/2 pages [4KB], 32768KB chunk

md4 : active raid5 sda8[0] sdd5[3] sdc6[2] sdb5[1]
      218636160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/2 pages [0KB], 32768KB chunk

md3 : active raid5 sda7[4] sdd3[3] sdc5[2] sdb3[1]
      218612160 blocks level 5, 64k chunk, algorithm 2 [4/3] [_UUU]
        resync=DELAYED
      bitmap: 2/2 pages [8KB], 32768KB chunk

md2 : active raid5 sda6[0] sdd2[3] sdc2[2] sdb2[1]
      30748032 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 1/1 pages [4KB], 32768KB chunk

md0 : active raid5 sda2[0] sdd1[2] sdc1[1]
      578048 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      bitmap: 0/1 pages [0KB], 32768KB chunk

md1 : active raid1 sdb1[0] sda5[1]
      289024 blocks [2/2] [UU]
      bitmap: 0/1 pages [0KB], 32768KB chunk

unused devices: <none>

------------------

/etc/mdadm/mdadm.conf:

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR jdc@uwo.ca

# definitions of existing MD arrays
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=6b8b4567:327b23c6:643c9869:66334873
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=ba493129:00074cd3:fee07e15:038135d5
ARRAY /dev/md2 level=raid5 num-devices=4 UUID=3dc9b50b:b9270472:9778d943:b967813b
ARRAY /dev/md3 level=raid5 num-devices=4 UUID=c4056d19:7b4bb550:44925b88:91d5bc8a
ARRAY /dev/md4 level=raid5 num-devices=4 UUID=d7c84402:210b78c7:556bbbc0:47df436c
ARRAY /dev/md5 level=raid5 num-devices=3 UUID=9effd43f:93ccc32d:899ca6c7:ea966964
ARRAY /dev/md6 level=raid1 num-devices=2 UUID=da17264f:be7e012d:85187211:fb0e2ebd


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: devices get kicked from RAID about once a month

[Dan Christensen]

rsivak@vshift.com writes:

> I would say that these are probably desktop type drives - not
> enterprise drives.

Right.

> In which case, what you're probably hitting are
> media errors - when a drive hits a bad sector.  A consumer level drive
> will try as hard as possible to retrieve the data - I'd say somewhere
> around 15 seconds, depending on the manufacturer.  This is often
> enough to get the drive kicked out of the array.

Could be.  I've included the SMART info for one of the drives below.
Most of the values look very good to me.  E.g. no re-allocated sectors
at all.  The only one that raises an eyebrow is:

195 Hardware_ECC_Recovered  0x001a   100   100   000    Old_age   Always       -       540903625

Do the stats below support the theory that these are timeouts due to
media errors?  Does anyone know if Samsung Spinpoint drives can be
tweaked to not try so hard to retrieve data?  Or is it possible to make
the kernel more tolerant of such timeouts?

Thanks,

Dan

smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Model Family:     SAMSUNG SpinPoint P120 series
Device Model:     SAMSUNG SP2504C
Serial Number:    S09QJ1UL210508
Firmware Version: VT100-33
User Capacity:    250,059,350,016 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   7
ATA Standard is:  ATA/ATAPI-7 T13 1532D revision 4a
Local Time is:    Wed Jun  2 11:23:40 2010 EDT

==> WARNING: May need -F samsung3 enabled; see manual for details.

SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84)	Offline data collection activity
					was suspended by an interrupting command from host.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		 (4965) seconds.
Offline data collection
capabilities: 			 (0x5b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 (  82) minutes.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   100   100   051    Pre-fail  Always       -       789
  3 Spin_Up_Time            0x0007   100   100   025    Pre-fail  Always       -       5952
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       101
  5 Reallocated_Sector_Ct   0x0033   253   253   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   253   253   051    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0025   253   253   015    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       34577
 10 Spin_Retry_Count        0x0033   253   253   051    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x0012   253   002   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       86
190 Airflow_Temperature_Cel 0x0022   142   106   000    Old_age   Always       -       32
194 Temperature_Celsius     0x0022   142   106   000    Old_age   Always       -       32
195 Hardware_ECC_Recovered  0x001a   100   100   000    Old_age   Always       -       540903625
196 Reallocated_Event_Count 0x0032   253   253   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   253   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   253   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x000a   100   100   000    Old_age   Always       -       0
201 Soft_Read_Error_Rate    0x000a   100   100   000    Old_age   Always       -       0
202 TA_Increase_Count       0x0032   253   253   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     34427         -
# 2  Extended offline    Completed without error       00%     34258         -
# 3  Extended offline    Completed without error       00%     34089         -
# 4  Extended offline    Completed without error       00%     34002         -
# 5  Short offline       Completed without error       00%     34001         -
# 6  Extended offline    Completed without error       00%     33921         -
# 7  Extended offline    Completed without error       00%     33754         -
# 8  Extended offline    Completed without error       00%     33586         -
# 9  Extended offline    Completed without error       00%     33417         -
#10  Extended offline    Completed without error       00%     33249         -
#11  Extended offline    Completed without error       00%     33077         -
#12  Extended offline    Completed without error       00%     32909         -
#13  Extended offline    Completed without error       00%     32740         -
#14  Extended offline    Completed without error       00%     32513         -
#15  Extended offline    Completed without error       00%     32344         -
#16  Extended offline    Completed without error       00%     32176         -
#17  Extended offline    Completed without error       00%     32066         -
#18  Short offline       Completed without error       00%     32064         -
#19  Extended offline    Completed without error       00%     32007         -
#20  Extended offline    Completed without error       00%     31839         -
#21  Extended offline    Completed without error       00%     31670         -

SMART Selective Self-Test Log Data Structure Revision Number (0) should be 1
SMART Selective self-test log data structure revision number 0
Warning: ATA Specification requires selective self-test log data structure revision number = 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Re: devices get kicked from RAID about once a month

[John Robinson]

On 02/06/2010 16:29, Dan Christensen wrote:
[...]
> Do the stats below support the theory that these are timeouts due to
> media errors?  Does anyone know if Samsung Spinpoint drives can be
> tweaked to not try so hard to retrieve data?  Or is it possible to make
> the kernel more tolerant of such timeouts?

My Samsung Spinpoint F1's can have TLER enabled using a more recent 
smartctl. It's not appeared as part of a formal release yet but a patch 
went in to r3065 in SVN: 
http://sourceforge.net/apps/trac/smartmontools/log/trunk/smartmontools

Cheers,

John.

Re: devices get kicked from RAID about once a month

[Dan Christensen]

John Robinson <john.robinson@anonymous.org.uk> writes:

> My Samsung Spinpoint F1's can have TLER enabled using a more recent
> smartctl. It's not appeared as part of a formal release yet but a
> patch went in to r3065 in SVN:
> http://sourceforge.net/apps/trac/smartmontools/log/trunk/smartmontools

Thanks.  I got the svn3077 from Debian testing, but it doesn't seem to
be supported with my drives:

# smartctl -l scterc /dev/sda
smartctl 5.40 2010-03-16 r3077 [i686-pc-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen,
http://smartmontools.sourceforge.net

Warning: device does not support SCT Commands

Any other suggestions?

Dan

Re: devices get kicked from RAID about once a month

[Bill Davidsen]

Dan Christensen wrote:
> John Robinson <john.robinson@anonymous.org.uk> writes:
>
>   
>> My Samsung Spinpoint F1's can have TLER enabled using a more recent
>> smartctl. It's not appeared as part of a formal release yet but a
>> patch went in to r3065 in SVN:
>> http://sourceforge.net/apps/trac/smartmontools/log/trunk/smartmontools
>>     
>
> Thanks.  I got the svn3077 from Debian testing, but it doesn't seem to
> be supported with my drives:
>
> # smartctl -l scterc /dev/sda
> smartctl 5.40 2010-03-16 r3077 [i686-pc-linux-gnu] (local build)
> Copyright (C) 2002-10 by Bruce Allen,
> http://smartmontools.sourceforge.net
>
> Warning: device does not support SCT Commands
>
> Any other suggestions?
>   

Presumably something shows in the logs, that's the next place to look.

-- 
Bill Davidsen <davidsen@tmr.com>
  "We can't solve today's problems by using the same thinking we
   used in creating them." - Einstein

Re: devices get kicked from RAID about once a month

[Dan Christensen]

Bill Davidsen <davidsen@tmr.com> writes:

> Presumably something shows in the logs, that's the next place to look.

I believe my drives simply don't support adjusting the time it takes to
try to recover a read (CCTL).  Or did you mean the logs at the time of
failure?  If so, I posted those in the original message.

Have there been any kernel changes since 2.6.28 that might improve
reliability of my set-up?  It would be nice if md could be told to try
again before giving up on the drives.

Thanks for all of the help so far!

Dan

Re: devices get kicked from RAID about once a month

[Stefan /*St0fF*/ Hübner]

-------- Original-Nachricht --------
Betreff: Re: devices get kicked from RAID about once a month
Datum: Wed, 02 Jun 2010 19:08:58 +0200
Von: Stefan /*St0fF*/ Hübner <st0ff@gmx.net>
Antwort an: st0ff@npl.de
An: Dan Christensen <jdc@uwo.ca>

Am 02.06.2010 18:33, schrieb Dan Christensen:
> John Robinson <john.robinson@anonymous.org.uk> writes:
> 
>> My Samsung Spinpoint F1's can have TLER enabled using a more recent
>> smartctl. It's not appeared as part of a formal release yet but a
>> patch went in to r3065 in SVN:
>> http://sourceforge.net/apps/trac/smartmontools/log/trunk/smartmontools
> 
> Thanks.  I got the svn3077 from Debian testing, but it doesn't seem to
> be supported with my drives:
> 
> # smartctl -l scterc /dev/sda
> smartctl 5.40 2010-03-16 r3077 [i686-pc-linux-gnu] (local build)
> Copyright (C) 2002-10 by Bruce Allen,
> http://smartmontools.sourceforge.net
> 
> Warning: device does not support SCT Commands

There you have it: the drives are not supporting SCT-ERC.  Which I can
say is true: I also own an SP2504C.
> 
> Any other suggestions?

Not really, it's up to Neil to export some sysfs-variable, where you
could tune how long a drive may take to respond to some command.

The "trying hard to get the data back" can take up to a few minutes
(somewhere I read about 2, somewhere else about 3 minutes).  And what
really is happening is: after some timeout the mdraid "thinks
correctly", that the drive cannot provide the requested sector.  To
prevent failure, it reconstructs the "missing" data and issues a write
request.  Unfortunately the drive still tries to reconstruct the data
itself and will not respond.  After those write requests failing, mdraid
drops the disk.

The ERC-setting is volatile.  You'd have to issue it on every reboot,
and on every hotswap.  But at first you'd have to get drives that
support this setting.

stefan
> 
> Dan
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: devices get kicked from RAID about once a month

[Miha Verlic]

On 06/02/2010 04:14 PM, Dan Christensen wrote:
> I tried disconnecting and re-connecting all four SATA cables, but the
> problem occurred again.  In fact, today *two* partitions were kicked out
> of their (different) raid devices.

Well the logs only show that separate hard drives stoped responding for
a while and kernel resetted sata link at that point.

Since it's happening on random drives, and even two at a time, I'd say
hard drives itself are not the problem. I'd replace power supply first.
I've had similar simptoms on one of my machines - it either crashed or
kicked one of drives out of the array. It happened randomly, but usually
once per month - who knows, maybe a spike in a grid, sudden cpu usage or
something similar caused it.

Anyway - bigger power supply solved the problem.

--
Miha

Re: devices get kicked from RAID about once a month

[Neil Brown]

On Wed, 02 Jun 2010 20:29:46 +0200
Stefan /*St0fF*/ Hübner <st0ff@gmx.net> wrote:

> > 
> > Any other suggestions?  
> 
> Not really, it's up to Neil to export some sysfs-variable, where you
> could tune how long a drive may take to respond to some command.
> 

Nope.  md doesn't do any timeouts.
You need to look for, or ask for, such variables at the scsi/sata layer.

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: devices get kicked from RAID about once a month

[Bill Davidsen]

Dan Christensen wrote:
> Bill Davidsen <davidsen@tmr.com> writes:
>
>   
>> Presumably something shows in the logs, that's the next place to look.
>>     
>
> I believe my drives simply don't support adjusting the time it takes to
> try to recover a read (CCTL).  Or did you mean the logs at the time of
> failure?  If so, I posted those in the original message.
>
>   
Those logs don't show any information useful to me which tells me how 
long md waited, and I'm not able to parse any of the res: information to 
gain clarity. It would be nice if someone can parse that, but I can't. 
On timeout an elapsed time output would be nice to indicate what the 
time limit is.

But I do have desktop drives in raid arrays, and they do spin down when 
not in use, and when I access the array after it's been unused I get a 
multi-second delay before my ls info comes back, so clearly there are 
some paths in the SATA handling which don't easily time out.

> Have there been any kernel changes since 2.6.28 that might improve
> reliability of my set-up?  It would be nice if md could be told to try
> again before giving up on the drives.
>
>   
Don't know, Neil might. I sure would like to see a timeout in ms in the 
/sys for the device and a flag for the array to not kick a drive for 
timeout until some number of consecutive timeouts have occurred. Note 
that I'm not suggesting some particular implementation, just some 
tunables. And if only one sector times out, a delayed reconstruct and 
rewrite might fix it. Again, a topic for discussion, not a "do thus" 
suggestion.

I would hope that a drive with multiple partitions would get the 
partitions kicked, not the whole drive at once. So one slow sector 
wouldn't take out multiple arrays.
> Thanks for all of the help so far!
>
> Dan
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>   


-- 
Bill Davidsen <davidsen@tmr.com>
  "We can't solve today's problems by using the same thinking we
   used in creating them." - Einstein

Re: devices get kicked from RAID about once a month

[Dan Christensen]

Bill Davidsen <davidsen@tmr.com> writes:

> Those logs don't show any information useful to me which tells me how
> long md waited, and I'm not able to parse any of the res: information
> to gain clarity. It would be nice if someone can parse that, but I
> can't. On timeout an elapsed time output would be nice to indicate
> what the time limit is.

I agree.  It would also be nice to know whether there was in fact a read
error at that time (in which case I may just replace the drives to avoid
this problem) or whether it was some other communications glitch (in
which case I may suspect the power supply, try a newer kernel, etc).
With the information at hand, I'm not sure how to fix this, and since
it often is a month or more between occurrences, trial and error is
not likely to help.

> I sure would like to see a timeout in ms [md?] in
> the /sys for the device and a flag for the array to not kick a drive
> for timeout until some number of consecutive timeouts have
> occurred. 

That could be useful.  And, as Neil said, if the SATA driver could be
told to use longer timeouts, that might help.  Neil, if you think that's
a good idea, maybe you could put the request in with the SATA folks?

> I would hope that a drive with multiple partitions would get the
> partitions kicked, not the whole drive at once. So one slow sector
> wouldn't take out multiple arrays.

Only the partition gets kicked out.  Yesterday, this saved me, since I
had timeouts on two drives in RAID5, but all the arrays stayed up because
the partitions didn't happen to be in the same array.

Dan

Re: devices get kicked from RAID about once a month

[Bill Davidsen]

Neil Brown wrote:
> On Wed, 02 Jun 2010 20:29:46 +0200
> Stefan /*St0fF*/ Hübner <st0ff@gmx.net> wrote:
>
>   
>>> Any other suggestions?  
>>>       
>> Not really, it's up to Neil to export some sysfs-variable, where you
>> could tune how long a drive may take to respond to some command.
>>
>>     
>
> Nope.  md doesn't do any timeouts.
>   

That's the problem. A timeout between getting the timeout status and 
trying the rewrite is really needed to have any hope of recovery.

If there were a write intent bitmap for the drive, perhaps the drive 
could enter some "may be recovering" state and writes, including the one 
to rewrite the sector, could be help off for some few minutes. I say 
that, knowing that there is at least some similar code working for 
network attached drives, which seem to survive a brief network issue.

Telling the user a write intent bitmap is needed and making use of it 
sound at all practical as a use for some existing code?

> You need to look for, or ask for, such variables at the scsi/sata layer.
>
>   
The need for a delay between timeout and rewrite

-- 
Bill Davidsen <davidsen@tmr.com>
  "We can't solve today's problems by using the same thinking we
   used in creating them." - Einstein


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: devices get kicked from RAID about once a month

[Neil Brown]

On Thu, 03 Jun 2010 12:47:39 -0400
Dan Christensen <jdc@uwo.ca> wrote:

> Bill Davidsen <davidsen@tmr.com> writes:
> 
> > Those logs don't show any information useful to me which tells me how
> > long md waited, and I'm not able to parse any of the res: information
> > to gain clarity. It would be nice if someone can parse that, but I
> > can't. On timeout an elapsed time output would be nice to indicate
> > what the time limit is.
> 
> I agree.  It would also be nice to know whether there was in fact a read
> error at that time (in which case I may just replace the drives to avoid
> this problem) or whether it was some other communications glitch (in
> which case I may suspect the power supply, try a newer kernel, etc).
> With the information at hand, I'm not sure how to fix this, and since
> it often is a month or more between occurrences, trial and error is
> not likely to help.
> 
> > I sure would like to see a timeout in ms [md?] in
> > the /sys for the device and a flag for the array to not kick a drive
> > for timeout until some number of consecutive timeouts have
> > occurred. 
> 
> That could be useful.  And, as Neil said, if the SATA driver could be
> told to use longer timeouts, that might help.  Neil, if you think that's
> a good idea, maybe you could put the request in with the SATA folks?

It might be a good idea.
Seeing you have the error logs, you have the border-line drives, you are in
the best position to test anything they suggest, and you have the strongest
motivation to see a resolution, I recommend you put in the request.  Email
details should be available in the MAINTAINERS file.

NeilBrown


> 
> > I would hope that a drive with multiple partitions would get the
> > partitions kicked, not the whole drive at once. So one slow sector
> > wouldn't take out multiple arrays.
> 
> Only the partition gets kicked out.  Yesterday, this saved me, since I
> had timeouts on two drives in RAID5, but all the arrays stayed up because
> the partitions didn't happen to be in the same array.
> 
> Dan
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: devices get kicked from RAID about once a month

[Dan Christensen]

Neil Brown <neilb@suse.de> writes:

> On Thu, 03 Jun 2010 12:47:39 -0400 Dan Christensen <jdc@uwo.ca> wrote:
>
>> That could be useful.  And, as Neil said, if the SATA driver could be
>> told to use longer timeouts, that might help.  Neil, if you think that's
>> a good idea, maybe you could put the request in with the SATA folks?
>
> It might be a good idea.

After thinking about it more, I'm not sure I fully understand the
situation.  

If I was able to turn on something like TLER on the drives, so read
errors failed more quickly, what would the raid layer do when it got
a read error? 

If the raid layer handles this in a clever way (and I recall some
discussions about this), e.g. by reconstructing the data and rewriting
the sector allowing the drive to remap it, then what I don't fully
understand is why it doesn't also do this when there is a timeout on a
read.  Is it because timeouts can indicate more serious problems?  Even
so, wouldn't it be reasonable for the raid layer to give the drive a
second chance before assuming it has failed?

These questions are motivated from the following logic.  Since it is
generally recognized that quicker read errors (e.g. TLER) are good
for drives in a raid array, *increasing* the SATA timeouts seems like it
is going in the wrong direction.  Wouldn't it be better to have short
timeouts, but have the raid layer treat a timeout less seriously?

Dan

Re: devices get kicked from RAID about once a month

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 2735 bytes --]

On Fri Jun 04, 2010 at 09:30:09AM -0400, Dan Christensen wrote:

> Neil Brown <neilb@suse.de> writes:
> 
> > On Thu, 03 Jun 2010 12:47:39 -0400 Dan Christensen <jdc@uwo.ca> wrote:
> >
> >> That could be useful.  And, as Neil said, if the SATA driver could be
> >> told to use longer timeouts, that might help.  Neil, if you think that's
> >> a good idea, maybe you could put the request in with the SATA folks?
> >
> > It might be a good idea.
> 
> After thinking about it more, I'm not sure I fully understand the
> situation.  
> 
> If I was able to turn on something like TLER on the drives, so read
> errors failed more quickly, what would the raid layer do when it got
> a read error? 
> 
It reconstructs the data and attempts a write.  A write failure will
then fail the drive.

> If the raid layer handles this in a clever way (and I recall some
> discussions about this), e.g. by reconstructing the data and rewriting
> the sector allowing the drive to remap it, then what I don't fully
> understand is why it doesn't also do this when there is a timeout on a
> read.  Is it because timeouts can indicate more serious problems?  Even
> so, wouldn't it be reasonable for the raid layer to give the drive a
> second chance before assuming it has failed?
> 
It does exactly the same on the read timeout.  The problem is that when
it sends the write, the drive is still busy attempting the read, so
ignores the write request (until it's free).  This then times out as
well, so the array assumes the drive has failed.

> These questions are motivated from the following logic.  Since it is
> generally recognized that quicker read errors (e.g. TLER) are good
> for drives in a raid array, *increasing* the SATA timeouts seems like it
> is going in the wrong direction.  Wouldn't it be better to have short
> timeouts, but have the raid layer treat a timeout less seriously?
> 
As has been stated, the RAID layer doesn't have any timeouts.  It's the
SCSI/ATA layer which is timing out the read/write and reporting a
failure to the RAID layer.  If the timeout at this level is increased
sufficiently then either the read will eventually succeed, or it'll
still fail but the write will then succeed (as the drive is no longer
busy) (or the write will fail and the disk is really failed).  The
downside is that the increased delay could run into timeouts at other
levels, but these are likely to be less severe than randomly failed
drives.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: devices get kicked from RAID about once a month

[Dan Christensen]

Robin Hill <robin@robinhill.me.uk> writes:

> On Fri Jun 04, 2010 at 09:30:09AM -0400, Dan Christensen wrote:
>
>> what would the raid layer do when it got a read error? 
>> 
> It reconstructs the data and attempts a write.  A write failure will
> then fail the drive.
[...]
> It does exactly the same on the read timeout.  The problem is that when
> it sends the write, the drive is still busy attempting the read, so
> ignores the write request (until it's free).  This then times out as
> well, so the array assumes the drive has failed.
>
>> These questions are motivated from the following logic.  Since it is
>> generally recognized that quicker read errors (e.g. TLER) are good
>> for drives in a raid array, *increasing* the SATA timeouts seems like it
>> is going in the wrong direction.  Wouldn't it be better to have short
>> timeouts, but have the raid layer treat a timeout less seriously?
>> 
> As has been stated, the RAID layer doesn't have any timeouts.  It's the
> SCSI/ATA layer which is timing out the read/write and reporting a
> failure to the RAID layer.  If the timeout at this level is increased
> sufficiently then either the read will eventually succeed, or it'll
> still fail but the write will then succeed (as the drive is no longer
> busy) (or the write will fail and the disk is really failed).

Ok, I now understand the idea here.  Even if the SATA timeout were
reduced, there's nothing the raid layer can do until the drive is
ready to respond again.  So it makes sense to work around this by
increasing the SATA timeouts.

Thanks for the clarification!

Dan