* Do I have a bad HDD?
@ 2011-07-31 13:05 Mathias Burén
2011-07-31 17:59 ` Mathias Burén
0 siblings, 1 reply; 9+ messages in thread
From: Mathias Burén @ 2011-07-31 13:05 UTC (permalink / raw)
To: Linux-RAID
Hi list,
Here's the output of my weekly script:
DEV EVENTS REALL PEND UNCORR CRC RAW ZONE END
sdb1 6158767 0 0 0 2 0 0
sdc1 6158767 0 0 0 0 0 0
sdd1 6158767 0 0 0 0 0 0
sde1 6158767 0 0 0 0 0 1
sdf1 6158767 0 0 0 0 47 6
sdg1 6158767 0 0 0 0 0 0
sdh1 6158767 0 6 0 0 340 3
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdf1[5] sdh1[6] sdg1[0] sde1[7] sdc1[3] sdd1[4] sdb1[1]
9751756800 blocks super 1.2 level 6, 64k chunk, algorithm 2
[7/7] [UUUUUUU]
unused devices: <none>
/dev/md0:
Version : 1.2
Creation Time : Tue Oct 19 08:58:41 2010
Raid Level : raid6
Array Size : 9751756800 (9300.00 GiB 9985.80 GB)
Used Dev Size : 1950351360 (1860.00 GiB 1997.16 GB)
Raid Devices : 7
Total Devices : 7
Persistence : Superblock is persistent
Update Time : Sun Jul 31 09:50:43 2011
State : clean
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : ion:0 (local to host ion)
UUID : e6595c64:b3ae90b3:f01133ac:3f402d20
Events : 6158767
Number Major Minor RaidDevice State
0 8 97 0 active sync /dev/sdg1
1 8 17 1 active sync /dev/sdb1
4 8 49 2 active sync /dev/sdd1
3 8 33 3 active sync /dev/sdc1
5 8 81 4 active sync /dev/sdf1
6 8 113 5 active sync /dev/sdh1
7 8 65 6 active sync /dev/sde1
Here's the SMART data for sdh:
smartctl 5.41 2011-06-09 r3365 [x86_64-linux-2.6.39-ck] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Model Family: SAMSUNG SpinPoint F4 EG (AFT)
Device Model: SAMSUNG HD204UI
Serial Number: S2HGJ1RZ800850
LU WWN Device Id: 5 0024e9 003f1ebc9
Firmware Version: 1AQ10003
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 6
Local Time is: Sun Jul 31 14:03:32 2011 IST
==> WARNING: Using smartmontools or hdparm with this
drive may result in data loss due to a firmware bug.
****** THIS DRIVE MAY OR MAY NOT BE AFFECTED! ******
Buggy and fixed firmware report same version number!
See the following web pages for details:
http://www.samsung.com/global/business/hdd/faqView.do?b2b_bbs_msg_id=386
http://sourceforge.net/apps/trac/smartmontools/wiki/SamsungF4EGBadBlocks
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 37) The self-test routine was interrupted
by the host with a hard or soft reset.
Total time to complete Offline
data collection: (20640) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 255) minutes.
SCT capabilities: (0x003f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 100 051 Pre-fail
Always - 340
2 Throughput_Performance 0x0026 055 053 000 Old_age
Always - 18989
3 Spin_Up_Time 0x0023 067 044 025 Pre-fail
Always - 10165
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 18
5 Reallocated_Sector_Ct 0x0033 252 252 010 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 252 252 051 Old_age
Always - 0
8 Seek_Time_Performance 0x0024 252 252 015 Old_age
Offline - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age
Always - 6447
10 Spin_Retry_Count 0x0032 252 252 051 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 252 252 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 20
181 Program_Fail_Cnt_Total 0x0022 100 100 000 Old_age
Always - 10117271
191 G-Sense_Error_Rate 0x0022 100 100 000 Old_age
Always - 1
192 Power-Off_Retract_Count 0x0022 252 252 000 Old_age
Always - 0
194 Temperature_Celsius 0x0002 064 057 000 Old_age
Always - 35 (Min/Max 16/43)
195 Hardware_ECC_Recovered 0x003a 100 100 000 Old_age
Always - 0
196 Reallocated_Event_Count 0x0032 252 252 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age
Always - 6
198 Offline_Uncorrectable 0x0030 252 252 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0036 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x002a 100 100 000 Old_age
Always - 3
223 Load_Retry_Count 0x0032 252 252 000 Old_age
Always - 0
225 Load_Cycle_Count 0x0032 100 100 000 Old_age
Always - 21
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Interrupted (host reset) 50% 6408 -
# 2 Extended offline Completed without error 00% 6317 -
# 3 Extended offline Completed without error 00% 6260 -
# 4 Extended offline Completed without error 00% 6232 -
# 5 Extended offline Completed without error 00% 6170 -
# 6 Extended offline Completed without error 00% 6064 -
# 7 Extended offline Completed without error 00% 6029 -
# 8 Extended offline Completed without error 00% 5898 -
# 9 Extended offline Aborted by host 60% 5893 -
#10 Extended offline Completed without error 00% 5728 -
#11 Extended offline Completed without error 00% 5706 -
#12 Extended offline Interrupted (host reset) 40% 5701 -
#13 Extended offline Interrupted (host reset) 90% 5666 -
#14 Extended offline Completed without error 00% 5560 -
#15 Extended offline Completed without error 00% 5527 -
#16 Extended offline Completed without error 00% 5392 -
#17 Extended offline Completed without error 00% 5357 -
#18 Extended offline Completed without error 00% 5250 -
#19 Extended offline Completed without error 00% 4272 -
#20 Extended offline Completed without error 00% 4017 -
#21 Extended offline Completed without error 00% 3935 -
Note: selective self-test log revision number (0) not 1 implies that
no selective self-test has ever been run
SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has
ever been run
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Interrupted [50% left] (0-65535)
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
It has 6 pending sectors. Why are they not reallocated? Can I force
this somehow? (a scrub did not reallocate them) Is this enough to
replace the HDD?
Thanks,
Mathias
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Do I have a bad HDD?
2011-07-31 13:05 Do I have a bad HDD? Mathias Burén
@ 2011-07-31 17:59 ` Mathias Burén
2011-07-31 18:05 ` Mathias Burén
0 siblings, 1 reply; 9+ messages in thread
From: Mathias Burén @ 2011-07-31 17:59 UTC (permalink / raw)
To: Linux-RAID
On 31 July 2011 14:05, Mathias Burén <mathias.buren@gmail.com> wrote:
> Hi list,
>
> Here's the output of my weekly script:
>
> DEV EVENTS REALL PEND UNCORR CRC RAW ZONE END
> sdb1 6158767 0 0 0 2 0 0
> sdc1 6158767 0 0 0 0 0 0
> sdd1 6158767 0 0 0 0 0 0
> sde1 6158767 0 0 0 0 0 1
> sdf1 6158767 0 0 0 0 47 6
> sdg1 6158767 0 0 0 0 0 0
> sdh1 6158767 0 6 0 0 340 3
>
>
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid6 sdf1[5] sdh1[6] sdg1[0] sde1[7] sdc1[3] sdd1[4] sdb1[1]
> 9751756800 blocks super 1.2 level 6, 64k chunk, algorithm 2
> [7/7] [UUUUUUU]
>
> unused devices: <none>
>
>
> /dev/md0:
> Version : 1.2
> Creation Time : Tue Oct 19 08:58:41 2010
> Raid Level : raid6
> Array Size : 9751756800 (9300.00 GiB 9985.80 GB)
> Used Dev Size : 1950351360 (1860.00 GiB 1997.16 GB)
> Raid Devices : 7
> Total Devices : 7
> Persistence : Superblock is persistent
>
> Update Time : Sun Jul 31 09:50:43 2011
> State : clean
> Active Devices : 7
> Working Devices : 7
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Name : ion:0 (local to host ion)
> UUID : e6595c64:b3ae90b3:f01133ac:3f402d20
> Events : 6158767
>
> Number Major Minor RaidDevice State
> 0 8 97 0 active sync /dev/sdg1
> 1 8 17 1 active sync /dev/sdb1
> 4 8 49 2 active sync /dev/sdd1
> 3 8 33 3 active sync /dev/sdc1
> 5 8 81 4 active sync /dev/sdf1
> 6 8 113 5 active sync /dev/sdh1
> 7 8 65 6 active sync /dev/sde1
>
> Here's the SMART data for sdh:
>
>
> smartctl 5.41 2011-06-09 r3365 [x86_64-linux-2.6.39-ck] (local build)
> Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
>
> === START OF INFORMATION SECTION ===
> Model Family: SAMSUNG SpinPoint F4 EG (AFT)
> Device Model: SAMSUNG HD204UI
> Serial Number: S2HGJ1RZ800850
> LU WWN Device Id: 5 0024e9 003f1ebc9
> Firmware Version: 1AQ10003
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Size: 512 bytes logical/physical
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: 8
> ATA Standard is: ATA-8-ACS revision 6
> Local Time is: Sun Jul 31 14:03:32 2011 IST
>
> ==> WARNING: Using smartmontools or hdparm with this
> drive may result in data loss due to a firmware bug.
> ****** THIS DRIVE MAY OR MAY NOT BE AFFECTED! ******
> Buggy and fixed firmware report same version number!
> See the following web pages for details:
> http://www.samsung.com/global/business/hdd/faqView.do?b2b_bbs_msg_id=386
> http://sourceforge.net/apps/trac/smartmontools/wiki/SamsungF4EGBadBlocks
>
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> General SMART Values:
> Offline data collection status: (0x82) Offline data collection activity
> was completed without error.
> Auto Offline Data Collection: Enabled.
> Self-test execution status: ( 37) The self-test routine was interrupted
> by the host with a hard or soft reset.
> Total time to complete Offline
> data collection: (20640) seconds.
> Offline data collection
> capabilities: (0x5b) SMART execute Offline immediate.
> Auto Offline data collection on/off support.
> Suspend Offline collection upon new
> command.
> Offline surface scan supported.
> Self-test supported.
> No Conveyance Self-test supported.
> Selective Self-test supported.
> SMART capabilities: (0x0003) Saves SMART data before entering
> power-saving mode.
> Supports SMART auto save timer.
> Error logging capability: (0x01) Error logging supported.
> General Purpose Logging supported.
> Short self-test routine
> recommended polling time: ( 2) minutes.
> Extended self-test routine
> recommended polling time: ( 255) minutes.
> SCT capabilities: (0x003f) SCT Status supported.
> SCT Error Recovery Control supported.
> SCT Feature Control supported.
> SCT Data Table supported.
>
> SMART Attributes Data Structure revision number: 16
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> UPDATED WHEN_FAILED RAW_VALUE
> 1 Raw_Read_Error_Rate 0x002f 100 100 051 Pre-fail
> Always - 340
> 2 Throughput_Performance 0x0026 055 053 000 Old_age
> Always - 18989
> 3 Spin_Up_Time 0x0023 067 044 025 Pre-fail
> Always - 10165
> 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> Always - 18
> 5 Reallocated_Sector_Ct 0x0033 252 252 010 Pre-fail
> Always - 0
> 7 Seek_Error_Rate 0x002e 252 252 051 Old_age
> Always - 0
> 8 Seek_Time_Performance 0x0024 252 252 015 Old_age
> Offline - 0
> 9 Power_On_Hours 0x0032 100 100 000 Old_age
> Always - 6447
> 10 Spin_Retry_Count 0x0032 252 252 051 Old_age
> Always - 0
> 11 Calibration_Retry_Count 0x0032 252 252 000 Old_age
> Always - 0
> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> Always - 20
> 181 Program_Fail_Cnt_Total 0x0022 100 100 000 Old_age
> Always - 10117271
> 191 G-Sense_Error_Rate 0x0022 100 100 000 Old_age
> Always - 1
> 192 Power-Off_Retract_Count 0x0022 252 252 000 Old_age
> Always - 0
> 194 Temperature_Celsius 0x0002 064 057 000 Old_age
> Always - 35 (Min/Max 16/43)
> 195 Hardware_ECC_Recovered 0x003a 100 100 000 Old_age
> Always - 0
> 196 Reallocated_Event_Count 0x0032 252 252 000 Old_age
> Always - 0
> 197 Current_Pending_Sector 0x0032 100 100 000 Old_age
> Always - 6
> 198 Offline_Uncorrectable 0x0030 252 252 000 Old_age
> Offline - 0
> 199 UDMA_CRC_Error_Count 0x0036 200 200 000 Old_age
> Always - 0
> 200 Multi_Zone_Error_Rate 0x002a 100 100 000 Old_age
> Always - 3
> 223 Load_Retry_Count 0x0032 252 252 000 Old_age
> Always - 0
> 225 Load_Cycle_Count 0x0032 100 100 000 Old_age
> Always - 21
>
> SMART Error Log Version: 1
> No Errors Logged
>
> SMART Self-test log structure revision number 1
> Num Test_Description Status Remaining
> LifeTime(hours) LBA_of_first_error
> # 1 Extended offline Interrupted (host reset) 50% 6408 -
> # 2 Extended offline Completed without error 00% 6317 -
> # 3 Extended offline Completed without error 00% 6260 -
> # 4 Extended offline Completed without error 00% 6232 -
> # 5 Extended offline Completed without error 00% 6170 -
> # 6 Extended offline Completed without error 00% 6064 -
> # 7 Extended offline Completed without error 00% 6029 -
> # 8 Extended offline Completed without error 00% 5898 -
> # 9 Extended offline Aborted by host 60% 5893 -
> #10 Extended offline Completed without error 00% 5728 -
> #11 Extended offline Completed without error 00% 5706 -
> #12 Extended offline Interrupted (host reset) 40% 5701 -
> #13 Extended offline Interrupted (host reset) 90% 5666 -
> #14 Extended offline Completed without error 00% 5560 -
> #15 Extended offline Completed without error 00% 5527 -
> #16 Extended offline Completed without error 00% 5392 -
> #17 Extended offline Completed without error 00% 5357 -
> #18 Extended offline Completed without error 00% 5250 -
> #19 Extended offline Completed without error 00% 4272 -
> #20 Extended offline Completed without error 00% 4017 -
> #21 Extended offline Completed without error 00% 3935 -
>
> Note: selective self-test log revision number (0) not 1 implies that
> no selective self-test has ever been run
> SMART Selective self-test log data structure revision number 0
> Note: revision number not 1 implies that no selective self-test has
> ever been run
> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> 1 0 0 Interrupted [50% left] (0-65535)
> 2 0 0 Not_testing
> 3 0 0 Not_testing
> 4 0 0 Not_testing
> 5 0 0 Not_testing
> Selective self-test flags (0x0):
> After scanning selected spans, do NOT read-scan remainder of disk.
> If Selective self-test is pending on power-up, resume after 0 minute delay.
>
>
> It has 6 pending sectors. Why are they not reallocated? Can I force
> this somehow? (a scrub did not reallocate them) Is this enough to
> replace the HDD?
>
> Thanks,
> Mathias
>
Uh oh, I did another scrub, and here's the status:
DEV EVENTS REALL PEND UNCORR CRC RAW ZONE END
sdb1 6158767 0 0 0 2 0 0
sdc1 6158767 0 0 0 0 0 0
sdd1 6158767 0 0 0 0 0 0
sde1 6158767 0 0 0 0 0 1
sdf1 6158768 0 0 0 0 47 6
sdg1 6158767 0 0 0 0 0 0
sdh1 6158767 0 8 1 0 341 3
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdh1[6] sdg1[0] sdf1[5] sde1[7] sdd1[4] sdb1[1] sdc1[3]
9751756800 blocks super 1.2 level 6, 64k chunk, algorithm 2
[7/7] [UUUUUUU]
unused devices: <none>
/dev/md0:
Version : 1.2
Creation Time : Tue Oct 19 08:58:41 2010
Raid Level : raid6
Array Size : 9751756800 (9300.00 GiB 9985.80 GB)
Used Dev Size : 1950351360 (1860.00 GiB 1997.16 GB)
Raid Devices : 7
Total Devices : 7
Persistence : Superblock is persistent
Update Time : Sun Jul 31 18:51:58 2011
State : clean
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : ion:0 (local to host ion)
UUID : e6595c64:b3ae90b3:f01133ac:3f402d20
Events : 6158767
Number Major Minor RaidDevice State
0 8 97 0 active sync /dev/sdg1
1 8 17 1 active sync /dev/sdb1
4 8 49 2 active sync /dev/sdd1
3 8 33 3 active sync /dev/sdc1
5 8 81 4 active sync /dev/sdf1
6 8 113 5 active sync /dev/sdh1
7 8 65 6 active sync /dev/sde1
smartctl 5.41 2011-06-09 r3365 [x86_64-linux-2.6.39-ck] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Model Family: SAMSUNG SpinPoint F4 EG (AFT)
Device Model: SAMSUNG HD204UI
Serial Number: S2HGJ1RZ800850
LU WWN Device Id: 5 0024e9 003f1ebc9
Firmware Version: 1AQ10003
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 6
Local Time is: Sun Jul 31 18:51:59 2011 IST
==> WARNING: Using smartmontools or hdparm with this
drive may result in data loss due to a firmware bug.
****** THIS DRIVE MAY OR MAY NOT BE AFFECTED! ******
Buggy and fixed firmware report same version number!
See the following web pages for details:
http://www.samsung.com/global/business/hdd/faqView.do?b2b_bbs_msg_id=386
http://sourceforge.net/apps/trac/smartmontools/wiki/SamsungF4EGBadBlocks
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 118) The previous self-test completed having
the read element of the test failed.
Total time to complete Offline
data collection: (20640) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 255) minutes.
SCT capabilities: (0x003f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 100 051 Pre-fail
Always - 341
2 Throughput_Performance 0x0026 055 053 000 Old_age
Always - 18989
3 Spin_Up_Time 0x0023 067 044 025 Pre-fail
Always - 10165
4 Start_Stop_Count 0x0032 100 100 000 Old_age
Always - 18
5 Reallocated_Sector_Ct 0x0033 252 252 010 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 252 252 051 Old_age
Always - 0
8 Seek_Time_Performance 0x0024 252 252 015 Old_age
Offline - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age
Always - 6452
10 Spin_Retry_Count 0x0032 252 252 051 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 252 252 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age
Always - 20
181 Program_Fail_Cnt_Total 0x0022 100 100 000 Old_age
Always - 10121757
191 G-Sense_Error_Rate 0x0022 100 100 000 Old_age
Always - 1
192 Power-Off_Retract_Count 0x0022 252 252 000 Old_age
Always - 0
194 Temperature_Celsius 0x0002 064 057 000 Old_age
Always - 31 (Min/Max 16/43)
195 Hardware_ECC_Recovered 0x003a 100 100 000 Old_age
Always - 0
196 Reallocated_Event_Count 0x0032 252 252 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age
Always - 8
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age
Offline - 1
199 UDMA_CRC_Error_Count 0x0036 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x002a 100 100 000 Old_age
Always - 3
223 Load_Retry_Count 0x0032 252 252 000 Old_age
Always - 0
225 Load_Cycle_Count 0x0032 100 100 000 Old_age
Always - 21
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 60% 6452
1519520304
# 2 Extended offline Interrupted (host reset) 50% 6408 -
# 3 Extended offline Completed without error 00% 6317 -
# 4 Extended offline Completed without error 00% 6260 -
# 5 Extended offline Completed without error 00% 6232 -
# 6 Extended offline Completed without error 00% 6170 -
# 7 Extended offline Completed without error 00% 6064 -
# 8 Extended offline Completed without error 00% 6029 -
# 9 Extended offline Completed without error 00% 5898 -
#10 Extended offline Aborted by host 60% 5893 -
#11 Extended offline Completed without error 00% 5728 -
#12 Extended offline Completed without error 00% 5706 -
#13 Extended offline Interrupted (host reset) 40% 5701 -
#14 Extended offline Interrupted (host reset) 90% 5666 -
#15 Extended offline Completed without error 00% 5560 -
#16 Extended offline Completed without error 00% 5527 -
#17 Extended offline Completed without error 00% 5392 -
#18 Extended offline Completed without error 00% 5357 -
#19 Extended offline Completed without error 00% 5250 -
#20 Extended offline Completed without error 00% 4272 -
#21 Extended offline Completed without error 00% 4017 -
Note: selective self-test log revision number (0) not 1 implies that
no selective self-test has ever been run
SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has
ever been run
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Completed_read_failure [60% left] (0-65535)
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
:( I do have a bad HDD. RMA is already in progress, I need to take out
the drive and ship it to Samsung in Holland. Will print labels at work
on Tuesday.
Questions:
* How do I remove this HDD without causing damage to the array? Is
this the correct way?:
mdadm --manage /dev/md0 --fail /dev/sdh1 # fail the device
mdadm --manage /dev/md0 --remove /dev/sdh1 # remove the device
* (shut down the system gracefully)
* (remove the HDD)
* (install new HDD)
* (start system)
sfdisk -d /dev/sde | sfdisk /dev/sdh # partition the new HDD
mdadm --manage /dev/md0 --add /dev/sdh1 # add the partition to the array
* After removing the HDD, should I do another scrub?
Thanks a lot in advance!
/Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Do I have a bad HDD?
2011-07-31 17:59 ` Mathias Burén
@ 2011-07-31 18:05 ` Mathias Burén
2011-07-31 18:14 ` Mathias Burén
0 siblings, 1 reply; 9+ messages in thread
From: Mathias Burén @ 2011-07-31 18:05 UTC (permalink / raw)
To: Linux-RAID
On 31 July 2011 18:59, Mathias Burén <mathias.buren@gmail.com> wrote:
> On 31 July 2011 14:05, Mathias Burén <mathias.buren@gmail.com> wrote:
>> Hi list,
>>
>> Here's the output of my weekly script:
>>
>> DEV EVENTS REALL PEND UNCORR CRC RAW ZONE END
>> sdb1 6158767 0 0 0 2 0 0
>> sdc1 6158767 0 0 0 0 0 0
>> sdd1 6158767 0 0 0 0 0 0
>> sde1 6158767 0 0 0 0 0 1
>> sdf1 6158767 0 0 0 0 47 6
>> sdg1 6158767 0 0 0 0 0 0
>> sdh1 6158767 0 6 0 0 340 3
>>
>>
>> Personalities : [raid6] [raid5] [raid4]
>> md0 : active raid6 sdf1[5] sdh1[6] sdg1[0] sde1[7] sdc1[3] sdd1[4] sdb1[1]
>> 9751756800 blocks super 1.2 level 6, 64k chunk, algorithm 2
>> [7/7] [UUUUUUU]
>>
>> unused devices: <none>
>>
>>
>> /dev/md0:
>> Version : 1.2
>> Creation Time : Tue Oct 19 08:58:41 2010
>> Raid Level : raid6
>> Array Size : 9751756800 (9300.00 GiB 9985.80 GB)
>> Used Dev Size : 1950351360 (1860.00 GiB 1997.16 GB)
>> Raid Devices : 7
>> Total Devices : 7
>> Persistence : Superblock is persistent
>>
>> Update Time : Sun Jul 31 09:50:43 2011
>> State : clean
>> Active Devices : 7
>> Working Devices : 7
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> Layout : left-symmetric
>> Chunk Size : 64K
>>
>> Name : ion:0 (local to host ion)
>> UUID : e6595c64:b3ae90b3:f01133ac:3f402d20
>> Events : 6158767
>>
>> Number Major Minor RaidDevice State
>> 0 8 97 0 active sync /dev/sdg1
>> 1 8 17 1 active sync /dev/sdb1
>> 4 8 49 2 active sync /dev/sdd1
>> 3 8 33 3 active sync /dev/sdc1
>> 5 8 81 4 active sync /dev/sdf1
>> 6 8 113 5 active sync /dev/sdh1
>> 7 8 65 6 active sync /dev/sde1
>>
>> Here's the SMART data for sdh:
>>
>>
>> smartctl 5.41 2011-06-09 r3365 [x86_64-linux-2.6.39-ck] (local build)
>> Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
>>
>> === START OF INFORMATION SECTION ===
>> Model Family: SAMSUNG SpinPoint F4 EG (AFT)
>> Device Model: SAMSUNG HD204UI
>> Serial Number: S2HGJ1RZ800850
>> LU WWN Device Id: 5 0024e9 003f1ebc9
>> Firmware Version: 1AQ10003
>> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
>> Sector Size: 512 bytes logical/physical
>> Device is: In smartctl database [for details use: -P show]
>> ATA Version is: 8
>> ATA Standard is: ATA-8-ACS revision 6
>> Local Time is: Sun Jul 31 14:03:32 2011 IST
>>
>> ==> WARNING: Using smartmontools or hdparm with this
>> drive may result in data loss due to a firmware bug.
>> ****** THIS DRIVE MAY OR MAY NOT BE AFFECTED! ******
>> Buggy and fixed firmware report same version number!
>> See the following web pages for details:
>> http://www.samsung.com/global/business/hdd/faqView.do?b2b_bbs_msg_id=386
>> http://sourceforge.net/apps/trac/smartmontools/wiki/SamsungF4EGBadBlocks
>>
>> SMART support is: Available - device has SMART capability.
>> SMART support is: Enabled
>>
>> === START OF READ SMART DATA SECTION ===
>> SMART overall-health self-assessment test result: PASSED
>>
>> General SMART Values:
>> Offline data collection status: (0x82) Offline data collection activity
>> was completed without error.
>> Auto Offline Data Collection: Enabled.
>> Self-test execution status: ( 37) The self-test routine was interrupted
>> by the host with a hard or soft reset.
>> Total time to complete Offline
>> data collection: (20640) seconds.
>> Offline data collection
>> capabilities: (0x5b) SMART execute Offline immediate.
>> Auto Offline data collection on/off support.
>> Suspend Offline collection upon new
>> command.
>> Offline surface scan supported.
>> Self-test supported.
>> No Conveyance Self-test supported.
>> Selective Self-test supported.
>> SMART capabilities: (0x0003) Saves SMART data before entering
>> power-saving mode.
>> Supports SMART auto save timer.
>> Error logging capability: (0x01) Error logging supported.
>> General Purpose Logging supported.
>> Short self-test routine
>> recommended polling time: ( 2) minutes.
>> Extended self-test routine
>> recommended polling time: ( 255) minutes.
>> SCT capabilities: (0x003f) SCT Status supported.
>> SCT Error Recovery Control supported.
>> SCT Feature Control supported.
>> SCT Data Table supported.
>>
>> SMART Attributes Data Structure revision number: 16
>> Vendor Specific SMART Attributes with Thresholds:
>> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
>> UPDATED WHEN_FAILED RAW_VALUE
>> 1 Raw_Read_Error_Rate 0x002f 100 100 051 Pre-fail
>> Always - 340
>> 2 Throughput_Performance 0x0026 055 053 000 Old_age
>> Always - 18989
>> 3 Spin_Up_Time 0x0023 067 044 025 Pre-fail
>> Always - 10165
>> 4 Start_Stop_Count 0x0032 100 100 000 Old_age
>> Always - 18
>> 5 Reallocated_Sector_Ct 0x0033 252 252 010 Pre-fail
>> Always - 0
>> 7 Seek_Error_Rate 0x002e 252 252 051 Old_age
>> Always - 0
>> 8 Seek_Time_Performance 0x0024 252 252 015 Old_age
>> Offline - 0
>> 9 Power_On_Hours 0x0032 100 100 000 Old_age
>> Always - 6447
>> 10 Spin_Retry_Count 0x0032 252 252 051 Old_age
>> Always - 0
>> 11 Calibration_Retry_Count 0x0032 252 252 000 Old_age
>> Always - 0
>> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
>> Always - 20
>> 181 Program_Fail_Cnt_Total 0x0022 100 100 000 Old_age
>> Always - 10117271
>> 191 G-Sense_Error_Rate 0x0022 100 100 000 Old_age
>> Always - 1
>> 192 Power-Off_Retract_Count 0x0022 252 252 000 Old_age
>> Always - 0
>> 194 Temperature_Celsius 0x0002 064 057 000 Old_age
>> Always - 35 (Min/Max 16/43)
>> 195 Hardware_ECC_Recovered 0x003a 100 100 000 Old_age
>> Always - 0
>> 196 Reallocated_Event_Count 0x0032 252 252 000 Old_age
>> Always - 0
>> 197 Current_Pending_Sector 0x0032 100 100 000 Old_age
>> Always - 6
>> 198 Offline_Uncorrectable 0x0030 252 252 000 Old_age
>> Offline - 0
>> 199 UDMA_CRC_Error_Count 0x0036 200 200 000 Old_age
>> Always - 0
>> 200 Multi_Zone_Error_Rate 0x002a 100 100 000 Old_age
>> Always - 3
>> 223 Load_Retry_Count 0x0032 252 252 000 Old_age
>> Always - 0
>> 225 Load_Cycle_Count 0x0032 100 100 000 Old_age
>> Always - 21
>>
>> SMART Error Log Version: 1
>> No Errors Logged
>>
>> SMART Self-test log structure revision number 1
>> Num Test_Description Status Remaining
>> LifeTime(hours) LBA_of_first_error
>> # 1 Extended offline Interrupted (host reset) 50% 6408 -
>> # 2 Extended offline Completed without error 00% 6317 -
>> # 3 Extended offline Completed without error 00% 6260 -
>> # 4 Extended offline Completed without error 00% 6232 -
>> # 5 Extended offline Completed without error 00% 6170 -
>> # 6 Extended offline Completed without error 00% 6064 -
>> # 7 Extended offline Completed without error 00% 6029 -
>> # 8 Extended offline Completed without error 00% 5898 -
>> # 9 Extended offline Aborted by host 60% 5893 -
>> #10 Extended offline Completed without error 00% 5728 -
>> #11 Extended offline Completed without error 00% 5706 -
>> #12 Extended offline Interrupted (host reset) 40% 5701 -
>> #13 Extended offline Interrupted (host reset) 90% 5666 -
>> #14 Extended offline Completed without error 00% 5560 -
>> #15 Extended offline Completed without error 00% 5527 -
>> #16 Extended offline Completed without error 00% 5392 -
>> #17 Extended offline Completed without error 00% 5357 -
>> #18 Extended offline Completed without error 00% 5250 -
>> #19 Extended offline Completed without error 00% 4272 -
>> #20 Extended offline Completed without error 00% 4017 -
>> #21 Extended offline Completed without error 00% 3935 -
>>
>> Note: selective self-test log revision number (0) not 1 implies that
>> no selective self-test has ever been run
>> SMART Selective self-test log data structure revision number 0
>> Note: revision number not 1 implies that no selective self-test has
>> ever been run
>> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
>> 1 0 0 Interrupted [50% left] (0-65535)
>> 2 0 0 Not_testing
>> 3 0 0 Not_testing
>> 4 0 0 Not_testing
>> 5 0 0 Not_testing
>> Selective self-test flags (0x0):
>> After scanning selected spans, do NOT read-scan remainder of disk.
>> If Selective self-test is pending on power-up, resume after 0 minute delay.
>>
>>
>> It has 6 pending sectors. Why are they not reallocated? Can I force
>> this somehow? (a scrub did not reallocate them) Is this enough to
>> replace the HDD?
>>
>> Thanks,
>> Mathias
>>
>
> Uh oh, I did another scrub, and here's the status:
>
> DEV EVENTS REALL PEND UNCORR CRC RAW ZONE END
> sdb1 6158767 0 0 0 2 0 0
> sdc1 6158767 0 0 0 0 0 0
> sdd1 6158767 0 0 0 0 0 0
> sde1 6158767 0 0 0 0 0 1
> sdf1 6158768 0 0 0 0 47 6
> sdg1 6158767 0 0 0 0 0 0
> sdh1 6158767 0 8 1 0 341 3
>
>
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid6 sdh1[6] sdg1[0] sdf1[5] sde1[7] sdd1[4] sdb1[1] sdc1[3]
> 9751756800 blocks super 1.2 level 6, 64k chunk, algorithm 2
> [7/7] [UUUUUUU]
>
> unused devices: <none>
>
>
> /dev/md0:
> Version : 1.2
> Creation Time : Tue Oct 19 08:58:41 2010
> Raid Level : raid6
> Array Size : 9751756800 (9300.00 GiB 9985.80 GB)
> Used Dev Size : 1950351360 (1860.00 GiB 1997.16 GB)
> Raid Devices : 7
> Total Devices : 7
> Persistence : Superblock is persistent
>
> Update Time : Sun Jul 31 18:51:58 2011
> State : clean
> Active Devices : 7
> Working Devices : 7
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Name : ion:0 (local to host ion)
> UUID : e6595c64:b3ae90b3:f01133ac:3f402d20
> Events : 6158767
>
> Number Major Minor RaidDevice State
> 0 8 97 0 active sync /dev/sdg1
> 1 8 17 1 active sync /dev/sdb1
> 4 8 49 2 active sync /dev/sdd1
> 3 8 33 3 active sync /dev/sdc1
> 5 8 81 4 active sync /dev/sdf1
> 6 8 113 5 active sync /dev/sdh1
> 7 8 65 6 active sync /dev/sde1
>
> smartctl 5.41 2011-06-09 r3365 [x86_64-linux-2.6.39-ck] (local build)
> Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
>
> === START OF INFORMATION SECTION ===
> Model Family: SAMSUNG SpinPoint F4 EG (AFT)
> Device Model: SAMSUNG HD204UI
> Serial Number: S2HGJ1RZ800850
> LU WWN Device Id: 5 0024e9 003f1ebc9
> Firmware Version: 1AQ10003
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Size: 512 bytes logical/physical
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: 8
> ATA Standard is: ATA-8-ACS revision 6
> Local Time is: Sun Jul 31 18:51:59 2011 IST
>
> ==> WARNING: Using smartmontools or hdparm with this
> drive may result in data loss due to a firmware bug.
> ****** THIS DRIVE MAY OR MAY NOT BE AFFECTED! ******
> Buggy and fixed firmware report same version number!
> See the following web pages for details:
> http://www.samsung.com/global/business/hdd/faqView.do?b2b_bbs_msg_id=386
> http://sourceforge.net/apps/trac/smartmontools/wiki/SamsungF4EGBadBlocks
>
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> General SMART Values:
> Offline data collection status: (0x80) Offline data collection activity
> was never started.
> Auto Offline Data Collection: Enabled.
> Self-test execution status: ( 118) The previous self-test completed having
> the read element of the test failed.
> Total time to complete Offline
> data collection: (20640) seconds.
> Offline data collection
> capabilities: (0x5b) SMART execute Offline immediate.
> Auto Offline data collection on/off support.
> Suspend Offline collection upon new
> command.
> Offline surface scan supported.
> Self-test supported.
> No Conveyance Self-test supported.
> Selective Self-test supported.
> SMART capabilities: (0x0003) Saves SMART data before entering
> power-saving mode.
> Supports SMART auto save timer.
> Error logging capability: (0x01) Error logging supported.
> General Purpose Logging supported.
> Short self-test routine
> recommended polling time: ( 2) minutes.
> Extended self-test routine
> recommended polling time: ( 255) minutes.
> SCT capabilities: (0x003f) SCT Status supported.
> SCT Error Recovery Control supported.
> SCT Feature Control supported.
> SCT Data Table supported.
>
> SMART Attributes Data Structure revision number: 16
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
> UPDATED WHEN_FAILED RAW_VALUE
> 1 Raw_Read_Error_Rate 0x002f 100 100 051 Pre-fail
> Always - 341
> 2 Throughput_Performance 0x0026 055 053 000 Old_age
> Always - 18989
> 3 Spin_Up_Time 0x0023 067 044 025 Pre-fail
> Always - 10165
> 4 Start_Stop_Count 0x0032 100 100 000 Old_age
> Always - 18
> 5 Reallocated_Sector_Ct 0x0033 252 252 010 Pre-fail
> Always - 0
> 7 Seek_Error_Rate 0x002e 252 252 051 Old_age
> Always - 0
> 8 Seek_Time_Performance 0x0024 252 252 015 Old_age
> Offline - 0
> 9 Power_On_Hours 0x0032 100 100 000 Old_age
> Always - 6452
> 10 Spin_Retry_Count 0x0032 252 252 051 Old_age
> Always - 0
> 11 Calibration_Retry_Count 0x0032 252 252 000 Old_age
> Always - 0
> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
> Always - 20
> 181 Program_Fail_Cnt_Total 0x0022 100 100 000 Old_age
> Always - 10121757
> 191 G-Sense_Error_Rate 0x0022 100 100 000 Old_age
> Always - 1
> 192 Power-Off_Retract_Count 0x0022 252 252 000 Old_age
> Always - 0
> 194 Temperature_Celsius 0x0002 064 057 000 Old_age
> Always - 31 (Min/Max 16/43)
> 195 Hardware_ECC_Recovered 0x003a 100 100 000 Old_age
> Always - 0
> 196 Reallocated_Event_Count 0x0032 252 252 000 Old_age
> Always - 0
> 197 Current_Pending_Sector 0x0032 100 100 000 Old_age
> Always - 8
> 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age
> Offline - 1
> 199 UDMA_CRC_Error_Count 0x0036 200 200 000 Old_age
> Always - 0
> 200 Multi_Zone_Error_Rate 0x002a 100 100 000 Old_age
> Always - 3
> 223 Load_Retry_Count 0x0032 252 252 000 Old_age
> Always - 0
> 225 Load_Cycle_Count 0x0032 100 100 000 Old_age
> Always - 21
>
> SMART Error Log Version: 1
> No Errors Logged
>
> SMART Self-test log structure revision number 1
> Num Test_Description Status Remaining
> LifeTime(hours) LBA_of_first_error
> # 1 Extended offline Completed: read failure 60% 6452
> 1519520304
> # 2 Extended offline Interrupted (host reset) 50% 6408 -
> # 3 Extended offline Completed without error 00% 6317 -
> # 4 Extended offline Completed without error 00% 6260 -
> # 5 Extended offline Completed without error 00% 6232 -
> # 6 Extended offline Completed without error 00% 6170 -
> # 7 Extended offline Completed without error 00% 6064 -
> # 8 Extended offline Completed without error 00% 6029 -
> # 9 Extended offline Completed without error 00% 5898 -
> #10 Extended offline Aborted by host 60% 5893 -
> #11 Extended offline Completed without error 00% 5728 -
> #12 Extended offline Completed without error 00% 5706 -
> #13 Extended offline Interrupted (host reset) 40% 5701 -
> #14 Extended offline Interrupted (host reset) 90% 5666 -
> #15 Extended offline Completed without error 00% 5560 -
> #16 Extended offline Completed without error 00% 5527 -
> #17 Extended offline Completed without error 00% 5392 -
> #18 Extended offline Completed without error 00% 5357 -
> #19 Extended offline Completed without error 00% 5250 -
> #20 Extended offline Completed without error 00% 4272 -
> #21 Extended offline Completed without error 00% 4017 -
>
> Note: selective self-test log revision number (0) not 1 implies that
> no selective self-test has ever been run
> SMART Selective self-test log data structure revision number 0
> Note: revision number not 1 implies that no selective self-test has
> ever been run
> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> 1 0 0 Completed_read_failure [60% left] (0-65535)
> 2 0 0 Not_testing
> 3 0 0 Not_testing
> 4 0 0 Not_testing
> 5 0 0 Not_testing
> Selective self-test flags (0x0):
> After scanning selected spans, do NOT read-scan remainder of disk.
> If Selective self-test is pending on power-up, resume after 0 minute delay.
>
> :( I do have a bad HDD. RMA is already in progress, I need to take out
> the drive and ship it to Samsung in Holland. Will print labels at work
> on Tuesday.
>
> Questions:
>
> * How do I remove this HDD without causing damage to the array? Is
> this the correct way?:
> mdadm --manage /dev/md0 --fail /dev/sdh1 # fail the device
> mdadm --manage /dev/md0 --remove /dev/sdh1 # remove the device
> * (shut down the system gracefully)
> * (remove the HDD)
> * (install new HDD)
> * (start system)
> sfdisk -d /dev/sde | sfdisk /dev/sdh # partition the new HDD
> mdadm --manage /dev/md0 --add /dev/sdh1 # add the partition to the array
>
> * After removing the HDD, should I do another scrub?
>
> Thanks a lot in advance!
>
> /Mathias
>
I think I need to hurry up:
[13957.348692] ata10.00: exception Emask 0x0 SAct 0x3 SErr 0x0 action 0x6 frozen
[13957.348704] ata10.00: failed command: READ FPDMA QUEUED
[13957.348716] ata10.00: cmd 60/08:00:00:ab:5a/00:00:e1:00:00/40 tag 0
ncq 4096 in
[13957.348719] res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask
0x4 (timeout)
[13957.348724] ata10.00: status: { DRDY }
[13957.348730] ata10.00: failed command: WRITE FPDMA QUEUED
[13957.348741] ata10.00: cmd 61/08:08:a8:e7:8d/00:00:2e:00:00/40 tag 1
ncq 4096 out
[13957.348743] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask
0x4 (timeout)
[13957.348749] ata10.00: status: { DRDY }
[13957.348759] ata10: hard resetting link
[13962.835319] ata10: link is slow to respond, please be patient (ready=0)
[13967.368679] ata10: SRST failed (errno=-16)
[13967.368693] ata10: hard resetting link
[13970.988699] ata10: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[13971.002394] ata10.00: configured for UDMA/133
[13971.002407] ata10.00: device reported invalid CHS sector 0
[13971.002413] ata10.00: device reported invalid CHS sector 0
[13971.002427] ata10: EH complete
[14001.358848] ata10.00: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x6 frozen
[14001.358862] ata10.00: failed command: READ FPDMA QUEUED
[14001.358887] ata10.00: cmd 60/08:08:00:ab:5a/00:00:e1:00:00/40 tag 1
ncq 4096 in
[14001.358890] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask
0x4 (timeout)
[14001.358898] ata10.00: status: { DRDY }
[14001.358913] ata10: hard resetting link
[14006.845324] ata10: link is slow to respond, please be patient (ready=0)
[14011.378656] ata10: SRST failed (errno=-16)
[14011.378669] ata10: hard resetting link
[14016.865323] ata10: link is slow to respond, please be patient (ready=0)
[14021.398640] ata10: SRST failed (errno=-16)
[14021.398652] ata10: hard resetting link
[14026.885310] ata10: link is slow to respond, please be patient (ready=0)
[14029.925349] ata10: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[14029.939048] ata10.00: configured for UDMA/133
[14029.939061] ata10.00: device reported invalid CHS sector 0
[14029.939078] ata10: EH complete
[14060.345358] ata10.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
[14060.345371] ata10.00: failed command: READ FPDMA QUEUED
[14060.345384] ata10.00: cmd 60/08:00:00:ab:5a/00:00:e1:00:00/40 tag 0
ncq 4096 in
[14060.345387] res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask
0x4 (timeout)
[14060.345394] ata10.00: status: { DRDY }
[14060.345407] ata10: hard resetting link
[14065.831985] ata10: link is slow to respond, please be patient (ready=0)
[14070.365333] ata10: SRST failed (errno=-16)
[14070.365345] ata10: hard resetting link
[14074.625347] ata10: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[14074.639043] ata10.00: configured for UDMA/133
[14074.639056] ata10.00: device reported invalid CHS sector 0
[14074.639088] ata10: EH complete
[14105.358687] ata10.00: NCQ disabled due to excessive errors
[14105.358700] ata10.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
[14105.358712] ata10.00: failed command: READ FPDMA QUEUED
[14105.358729] ata10.00: cmd 60/08:00:00:ab:5a/00:00:e1:00:00/40 tag 0
ncq 4096 in
[14105.358732] res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask
0x4 (timeout)
[14105.358741] ata10.00: status: { DRDY }
[14105.358754] ata10: hard resetting link
[14110.845314] ata10: link is slow to respond, please be patient (ready=0)
[14115.378674] ata10: SRST failed (errno=-16)
[14115.378689] ata10: hard resetting link
[14119.372023] ata10: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[14119.385704] ata10.00: configured for UDMA/133
[14119.385716] ata10.00: device reported invalid CHS sector 0
[14119.385743] ata10: EH complete
[14121.527814] ata10.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6
[14121.527824] ata10.00: edma_err_cause=00000084 pp_flags=00000001,
dev error, EDMA self-disable
[14121.527832] ata10.00: failed command: READ DMA EXT
[14121.527844] ata10.00: cmd 25/00:08:00:ab:5a/00:00:e1:00:00/e0 tag 0
dma 4096 in
[14121.527846] res 51/89:08:00:ab:5a/89:00:e1:00:00/e0 Emask
0x10 (ATA bus error)
[14121.527852] ata10.00: status: { DRDY ERR }
[14121.527857] ata10.00: error: { ICRC }
[14121.527867] ata10: hard resetting link
[14127.011973] ata10: link is slow to respond, please be patient (ready=0)
[14131.545295] ata10: SRST failed (errno=-16)
[14131.545307] ata10: hard resetting link
[14137.031984] ata10: link is slow to respond, please be patient (ready=0)
[14141.565306] ata10: SRST failed (errno=-16)
[14141.565317] ata10: hard resetting link
[14147.051993] ata10: link is slow to respond, please be patient (ready=0)
[14161.032035] INFO: task jbd2/dm-0-8:613 blocked for more than 120 seconds.
[14161.032044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[14161.032050] jbd2/dm-0-8 D ffffffff81823020 0 613 2 0x00000000
[14161.032060] ffff8800c91e11a0 0000000000000046 0000000000000000
ffff8800cf0697c0
[14161.032070] ffff8800cf069780 ffff8800cf069780 ffff8800c93bdfd8
ffff8800c93bdfd8
[14161.032079] ffff8800c91e13d8 0000000000004000 ffff880094ed19c0
ffff8800c97e0688
[14161.032087] Call Trace:
[14161.032106] [<ffffffff81204c67>] ? generic_make_request+0x2f7/0x570
[14161.032116] [<ffffffff81133200>] ? __wait_on_buffer+0x30/0x30
[14161.032124] [<ffffffff814262e7>] ? io_schedule+0x57/0x80
[14161.032131] [<ffffffff8113320a>] ? sleep_on_buffer+0xa/0x20
[14161.032137] [<ffffffff81426a2f>] ? __wait_on_bit+0x4f/0x80
[14161.032143] [<ffffffff81133200>] ? __wait_on_buffer+0x30/0x30
[14161.032150] [<ffffffff81426add>] ? out_of_line_wait_on_bit+0x7d/0xa0
[14161.032159] [<ffffffff81059300>] ? autoremove_wake_function+0x30/0x30
[14161.032168] [<ffffffff811b680e>] ?
jbd2_journal_commit_transaction+0x155e/0x16f0
[14161.032176] [<ffffffff810592d0>] ? abort_exclusive_wait+0xb0/0xb0
[14161.032183] [<ffffffff8142968e>] ? apic_timer_interrupt+0xe/0x20
[14161.032191] [<ffffffff811baddd>] ? kjournald2+0xad/0x210
[14161.032198] [<ffffffff810592d0>] ? abort_exclusive_wait+0xb0/0xb0
[14161.032205] [<ffffffff811bad30>] ? commit_timeout+0x10/0x10
[14161.032212] [<ffffffff81058a0f>] ? kthread+0x7f/0x90
[14161.032219] [<ffffffff81429a54>] ? kernel_thread_helper+0x4/0x10
[14161.032226] [<ffffffff81058990>] ? kthread_worker_fn+0x180/0x180
[14161.032233] [<ffffffff81429a50>] ? gs_change+0xb/0xb
[14161.032261] INFO: task squid:1659 blocked for more than 120 seconds.
[14161.032264] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[14161.032269] squid D ffffffff81823020 0 1659 1657 0x00000000
[14161.032277] ffff8800c9179d60 0000000000000082 ffff8800c97b0688
ffffffff812bf074
[14161.032285] ffff880012b818c0 ffffffff81823020 ffff8800c22c5fd8
ffff8800c22c5fd8
[14161.032293] ffff8800c9179f98 0000000000004000 ffff8800c22c5fd8
0000000000000000
[14161.032301] Call Trace:
:-/
Currently shutting down all daemons that access the filesystem on the
array, in attempt to umount the fs.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Do I have a bad HDD?
2011-07-31 18:05 ` Mathias Burén
@ 2011-07-31 18:14 ` Mathias Burén
2011-07-31 18:25 ` Johannes Truschnigg
0 siblings, 1 reply; 9+ messages in thread
From: Mathias Burén @ 2011-07-31 18:14 UTC (permalink / raw)
To: Linux-RAID
On 31 July 2011 19:05, Mathias Burén <mathias.buren@gmail.com> wrote:
> On 31 July 2011 18:59, Mathias Burén <mathias.buren@gmail.com> wrote:
>> On 31 July 2011 14:05, Mathias Burén <mathias.buren@gmail.com> wrote:
>>> Hi list,
>>>
>>> Here's the output of my weekly script:
>>>
>>> DEV EVENTS REALL PEND UNCORR CRC RAW ZONE END
>>> sdb1 6158767 0 0 0 2 0 0
>>> sdc1 6158767 0 0 0 0 0 0
>>> sdd1 6158767 0 0 0 0 0 0
>>> sde1 6158767 0 0 0 0 0 1
>>> sdf1 6158767 0 0 0 0 47 6
>>> sdg1 6158767 0 0 0 0 0 0
>>> sdh1 6158767 0 6 0 0 340 3
>>>
>>>
>>> Personalities : [raid6] [raid5] [raid4]
>>> md0 : active raid6 sdf1[5] sdh1[6] sdg1[0] sde1[7] sdc1[3] sdd1[4] sdb1[1]
>>> 9751756800 blocks super 1.2 level 6, 64k chunk, algorithm 2
>>> [7/7] [UUUUUUU]
>>>
>>> unused devices: <none>
>>>
>>>
>>> /dev/md0:
>>> Version : 1.2
>>> Creation Time : Tue Oct 19 08:58:41 2010
>>> Raid Level : raid6
>>> Array Size : 9751756800 (9300.00 GiB 9985.80 GB)
>>> Used Dev Size : 1950351360 (1860.00 GiB 1997.16 GB)
>>> Raid Devices : 7
>>> Total Devices : 7
>>> Persistence : Superblock is persistent
>>>
>>> Update Time : Sun Jul 31 09:50:43 2011
>>> State : clean
>>> Active Devices : 7
>>> Working Devices : 7
>>> Failed Devices : 0
>>> Spare Devices : 0
>>>
>>> Layout : left-symmetric
>>> Chunk Size : 64K
>>>
>>> Name : ion:0 (local to host ion)
>>> UUID : e6595c64:b3ae90b3:f01133ac:3f402d20
>>> Events : 6158767
>>>
>>> Number Major Minor RaidDevice State
>>> 0 8 97 0 active sync /dev/sdg1
>>> 1 8 17 1 active sync /dev/sdb1
>>> 4 8 49 2 active sync /dev/sdd1
>>> 3 8 33 3 active sync /dev/sdc1
>>> 5 8 81 4 active sync /dev/sdf1
>>> 6 8 113 5 active sync /dev/sdh1
>>> 7 8 65 6 active sync /dev/sde1
>>>
>>> Here's the SMART data for sdh:
>>>
>>>
>>> smartctl 5.41 2011-06-09 r3365 [x86_64-linux-2.6.39-ck] (local build)
>>> Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
>>>
>>> === START OF INFORMATION SECTION ===
>>> Model Family: SAMSUNG SpinPoint F4 EG (AFT)
>>> Device Model: SAMSUNG HD204UI
>>> Serial Number: S2HGJ1RZ800850
>>> LU WWN Device Id: 5 0024e9 003f1ebc9
>>> Firmware Version: 1AQ10003
>>> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
>>> Sector Size: 512 bytes logical/physical
>>> Device is: In smartctl database [for details use: -P show]
>>> ATA Version is: 8
>>> ATA Standard is: ATA-8-ACS revision 6
>>> Local Time is: Sun Jul 31 14:03:32 2011 IST
>>>
>>> ==> WARNING: Using smartmontools or hdparm with this
>>> drive may result in data loss due to a firmware bug.
>>> ****** THIS DRIVE MAY OR MAY NOT BE AFFECTED! ******
>>> Buggy and fixed firmware report same version number!
>>> See the following web pages for details:
>>> http://www.samsung.com/global/business/hdd/faqView.do?b2b_bbs_msg_id=386
>>> http://sourceforge.net/apps/trac/smartmontools/wiki/SamsungF4EGBadBlocks
>>>
>>> SMART support is: Available - device has SMART capability.
>>> SMART support is: Enabled
>>>
>>> === START OF READ SMART DATA SECTION ===
>>> SMART overall-health self-assessment test result: PASSED
>>>
>>> General SMART Values:
>>> Offline data collection status: (0x82) Offline data collection activity
>>> was completed without error.
>>> Auto Offline Data Collection: Enabled.
>>> Self-test execution status: ( 37) The self-test routine was interrupted
>>> by the host with a hard or soft reset.
>>> Total time to complete Offline
>>> data collection: (20640) seconds.
>>> Offline data collection
>>> capabilities: (0x5b) SMART execute Offline immediate.
>>> Auto Offline data collection on/off support.
>>> Suspend Offline collection upon new
>>> command.
>>> Offline surface scan supported.
>>> Self-test supported.
>>> No Conveyance Self-test supported.
>>> Selective Self-test supported.
>>> SMART capabilities: (0x0003) Saves SMART data before entering
>>> power-saving mode.
>>> Supports SMART auto save timer.
>>> Error logging capability: (0x01) Error logging supported.
>>> General Purpose Logging supported.
>>> Short self-test routine
>>> recommended polling time: ( 2) minutes.
>>> Extended self-test routine
>>> recommended polling time: ( 255) minutes.
>>> SCT capabilities: (0x003f) SCT Status supported.
>>> SCT Error Recovery Control supported.
>>> SCT Feature Control supported.
>>> SCT Data Table supported.
>>>
>>> SMART Attributes Data Structure revision number: 16
>>> Vendor Specific SMART Attributes with Thresholds:
>>> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
>>> UPDATED WHEN_FAILED RAW_VALUE
>>> 1 Raw_Read_Error_Rate 0x002f 100 100 051 Pre-fail
>>> Always - 340
>>> 2 Throughput_Performance 0x0026 055 053 000 Old_age
>>> Always - 18989
>>> 3 Spin_Up_Time 0x0023 067 044 025 Pre-fail
>>> Always - 10165
>>> 4 Start_Stop_Count 0x0032 100 100 000 Old_age
>>> Always - 18
>>> 5 Reallocated_Sector_Ct 0x0033 252 252 010 Pre-fail
>>> Always - 0
>>> 7 Seek_Error_Rate 0x002e 252 252 051 Old_age
>>> Always - 0
>>> 8 Seek_Time_Performance 0x0024 252 252 015 Old_age
>>> Offline - 0
>>> 9 Power_On_Hours 0x0032 100 100 000 Old_age
>>> Always - 6447
>>> 10 Spin_Retry_Count 0x0032 252 252 051 Old_age
>>> Always - 0
>>> 11 Calibration_Retry_Count 0x0032 252 252 000 Old_age
>>> Always - 0
>>> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
>>> Always - 20
>>> 181 Program_Fail_Cnt_Total 0x0022 100 100 000 Old_age
>>> Always - 10117271
>>> 191 G-Sense_Error_Rate 0x0022 100 100 000 Old_age
>>> Always - 1
>>> 192 Power-Off_Retract_Count 0x0022 252 252 000 Old_age
>>> Always - 0
>>> 194 Temperature_Celsius 0x0002 064 057 000 Old_age
>>> Always - 35 (Min/Max 16/43)
>>> 195 Hardware_ECC_Recovered 0x003a 100 100 000 Old_age
>>> Always - 0
>>> 196 Reallocated_Event_Count 0x0032 252 252 000 Old_age
>>> Always - 0
>>> 197 Current_Pending_Sector 0x0032 100 100 000 Old_age
>>> Always - 6
>>> 198 Offline_Uncorrectable 0x0030 252 252 000 Old_age
>>> Offline - 0
>>> 199 UDMA_CRC_Error_Count 0x0036 200 200 000 Old_age
>>> Always - 0
>>> 200 Multi_Zone_Error_Rate 0x002a 100 100 000 Old_age
>>> Always - 3
>>> 223 Load_Retry_Count 0x0032 252 252 000 Old_age
>>> Always - 0
>>> 225 Load_Cycle_Count 0x0032 100 100 000 Old_age
>>> Always - 21
>>>
>>> SMART Error Log Version: 1
>>> No Errors Logged
>>>
>>> SMART Self-test log structure revision number 1
>>> Num Test_Description Status Remaining
>>> LifeTime(hours) LBA_of_first_error
>>> # 1 Extended offline Interrupted (host reset) 50% 6408 -
>>> # 2 Extended offline Completed without error 00% 6317 -
>>> # 3 Extended offline Completed without error 00% 6260 -
>>> # 4 Extended offline Completed without error 00% 6232 -
>>> # 5 Extended offline Completed without error 00% 6170 -
>>> # 6 Extended offline Completed without error 00% 6064 -
>>> # 7 Extended offline Completed without error 00% 6029 -
>>> # 8 Extended offline Completed without error 00% 5898 -
>>> # 9 Extended offline Aborted by host 60% 5893 -
>>> #10 Extended offline Completed without error 00% 5728 -
>>> #11 Extended offline Completed without error 00% 5706 -
>>> #12 Extended offline Interrupted (host reset) 40% 5701 -
>>> #13 Extended offline Interrupted (host reset) 90% 5666 -
>>> #14 Extended offline Completed without error 00% 5560 -
>>> #15 Extended offline Completed without error 00% 5527 -
>>> #16 Extended offline Completed without error 00% 5392 -
>>> #17 Extended offline Completed without error 00% 5357 -
>>> #18 Extended offline Completed without error 00% 5250 -
>>> #19 Extended offline Completed without error 00% 4272 -
>>> #20 Extended offline Completed without error 00% 4017 -
>>> #21 Extended offline Completed without error 00% 3935 -
>>>
>>> Note: selective self-test log revision number (0) not 1 implies that
>>> no selective self-test has ever been run
>>> SMART Selective self-test log data structure revision number 0
>>> Note: revision number not 1 implies that no selective self-test has
>>> ever been run
>>> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
>>> 1 0 0 Interrupted [50% left] (0-65535)
>>> 2 0 0 Not_testing
>>> 3 0 0 Not_testing
>>> 4 0 0 Not_testing
>>> 5 0 0 Not_testing
>>> Selective self-test flags (0x0):
>>> After scanning selected spans, do NOT read-scan remainder of disk.
>>> If Selective self-test is pending on power-up, resume after 0 minute delay.
>>>
>>>
>>> It has 6 pending sectors. Why are they not reallocated? Can I force
>>> this somehow? (a scrub did not reallocate them) Is this enough to
>>> replace the HDD?
>>>
>>> Thanks,
>>> Mathias
>>>
>>
>> Uh oh, I did another scrub, and here's the status:
>>
>> DEV EVENTS REALL PEND UNCORR CRC RAW ZONE END
>> sdb1 6158767 0 0 0 2 0 0
>> sdc1 6158767 0 0 0 0 0 0
>> sdd1 6158767 0 0 0 0 0 0
>> sde1 6158767 0 0 0 0 0 1
>> sdf1 6158768 0 0 0 0 47 6
>> sdg1 6158767 0 0 0 0 0 0
>> sdh1 6158767 0 8 1 0 341 3
>>
>>
>> Personalities : [raid6] [raid5] [raid4]
>> md0 : active raid6 sdh1[6] sdg1[0] sdf1[5] sde1[7] sdd1[4] sdb1[1] sdc1[3]
>> 9751756800 blocks super 1.2 level 6, 64k chunk, algorithm 2
>> [7/7] [UUUUUUU]
>>
>> unused devices: <none>
>>
>>
>> /dev/md0:
>> Version : 1.2
>> Creation Time : Tue Oct 19 08:58:41 2010
>> Raid Level : raid6
>> Array Size : 9751756800 (9300.00 GiB 9985.80 GB)
>> Used Dev Size : 1950351360 (1860.00 GiB 1997.16 GB)
>> Raid Devices : 7
>> Total Devices : 7
>> Persistence : Superblock is persistent
>>
>> Update Time : Sun Jul 31 18:51:58 2011
>> State : clean
>> Active Devices : 7
>> Working Devices : 7
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> Layout : left-symmetric
>> Chunk Size : 64K
>>
>> Name : ion:0 (local to host ion)
>> UUID : e6595c64:b3ae90b3:f01133ac:3f402d20
>> Events : 6158767
>>
>> Number Major Minor RaidDevice State
>> 0 8 97 0 active sync /dev/sdg1
>> 1 8 17 1 active sync /dev/sdb1
>> 4 8 49 2 active sync /dev/sdd1
>> 3 8 33 3 active sync /dev/sdc1
>> 5 8 81 4 active sync /dev/sdf1
>> 6 8 113 5 active sync /dev/sdh1
>> 7 8 65 6 active sync /dev/sde1
>>
>> smartctl 5.41 2011-06-09 r3365 [x86_64-linux-2.6.39-ck] (local build)
>> Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
>>
>> === START OF INFORMATION SECTION ===
>> Model Family: SAMSUNG SpinPoint F4 EG (AFT)
>> Device Model: SAMSUNG HD204UI
>> Serial Number: S2HGJ1RZ800850
>> LU WWN Device Id: 5 0024e9 003f1ebc9
>> Firmware Version: 1AQ10003
>> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
>> Sector Size: 512 bytes logical/physical
>> Device is: In smartctl database [for details use: -P show]
>> ATA Version is: 8
>> ATA Standard is: ATA-8-ACS revision 6
>> Local Time is: Sun Jul 31 18:51:59 2011 IST
>>
>> ==> WARNING: Using smartmontools or hdparm with this
>> drive may result in data loss due to a firmware bug.
>> ****** THIS DRIVE MAY OR MAY NOT BE AFFECTED! ******
>> Buggy and fixed firmware report same version number!
>> See the following web pages for details:
>> http://www.samsung.com/global/business/hdd/faqView.do?b2b_bbs_msg_id=386
>> http://sourceforge.net/apps/trac/smartmontools/wiki/SamsungF4EGBadBlocks
>>
>> SMART support is: Available - device has SMART capability.
>> SMART support is: Enabled
>>
>> === START OF READ SMART DATA SECTION ===
>> SMART overall-health self-assessment test result: PASSED
>>
>> General SMART Values:
>> Offline data collection status: (0x80) Offline data collection activity
>> was never started.
>> Auto Offline Data Collection: Enabled.
>> Self-test execution status: ( 118) The previous self-test completed having
>> the read element of the test failed.
>> Total time to complete Offline
>> data collection: (20640) seconds.
>> Offline data collection
>> capabilities: (0x5b) SMART execute Offline immediate.
>> Auto Offline data collection on/off support.
>> Suspend Offline collection upon new
>> command.
>> Offline surface scan supported.
>> Self-test supported.
>> No Conveyance Self-test supported.
>> Selective Self-test supported.
>> SMART capabilities: (0x0003) Saves SMART data before entering
>> power-saving mode.
>> Supports SMART auto save timer.
>> Error logging capability: (0x01) Error logging supported.
>> General Purpose Logging supported.
>> Short self-test routine
>> recommended polling time: ( 2) minutes.
>> Extended self-test routine
>> recommended polling time: ( 255) minutes.
>> SCT capabilities: (0x003f) SCT Status supported.
>> SCT Error Recovery Control supported.
>> SCT Feature Control supported.
>> SCT Data Table supported.
>>
>> SMART Attributes Data Structure revision number: 16
>> Vendor Specific SMART Attributes with Thresholds:
>> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
>> UPDATED WHEN_FAILED RAW_VALUE
>> 1 Raw_Read_Error_Rate 0x002f 100 100 051 Pre-fail
>> Always - 341
>> 2 Throughput_Performance 0x0026 055 053 000 Old_age
>> Always - 18989
>> 3 Spin_Up_Time 0x0023 067 044 025 Pre-fail
>> Always - 10165
>> 4 Start_Stop_Count 0x0032 100 100 000 Old_age
>> Always - 18
>> 5 Reallocated_Sector_Ct 0x0033 252 252 010 Pre-fail
>> Always - 0
>> 7 Seek_Error_Rate 0x002e 252 252 051 Old_age
>> Always - 0
>> 8 Seek_Time_Performance 0x0024 252 252 015 Old_age
>> Offline - 0
>> 9 Power_On_Hours 0x0032 100 100 000 Old_age
>> Always - 6452
>> 10 Spin_Retry_Count 0x0032 252 252 051 Old_age
>> Always - 0
>> 11 Calibration_Retry_Count 0x0032 252 252 000 Old_age
>> Always - 0
>> 12 Power_Cycle_Count 0x0032 100 100 000 Old_age
>> Always - 20
>> 181 Program_Fail_Cnt_Total 0x0022 100 100 000 Old_age
>> Always - 10121757
>> 191 G-Sense_Error_Rate 0x0022 100 100 000 Old_age
>> Always - 1
>> 192 Power-Off_Retract_Count 0x0022 252 252 000 Old_age
>> Always - 0
>> 194 Temperature_Celsius 0x0002 064 057 000 Old_age
>> Always - 31 (Min/Max 16/43)
>> 195 Hardware_ECC_Recovered 0x003a 100 100 000 Old_age
>> Always - 0
>> 196 Reallocated_Event_Count 0x0032 252 252 000 Old_age
>> Always - 0
>> 197 Current_Pending_Sector 0x0032 100 100 000 Old_age
>> Always - 8
>> 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age
>> Offline - 1
>> 199 UDMA_CRC_Error_Count 0x0036 200 200 000 Old_age
>> Always - 0
>> 200 Multi_Zone_Error_Rate 0x002a 100 100 000 Old_age
>> Always - 3
>> 223 Load_Retry_Count 0x0032 252 252 000 Old_age
>> Always - 0
>> 225 Load_Cycle_Count 0x0032 100 100 000 Old_age
>> Always - 21
>>
>> SMART Error Log Version: 1
>> No Errors Logged
>>
>> SMART Self-test log structure revision number 1
>> Num Test_Description Status Remaining
>> LifeTime(hours) LBA_of_first_error
>> # 1 Extended offline Completed: read failure 60% 6452
>> 1519520304
>> # 2 Extended offline Interrupted (host reset) 50% 6408 -
>> # 3 Extended offline Completed without error 00% 6317 -
>> # 4 Extended offline Completed without error 00% 6260 -
>> # 5 Extended offline Completed without error 00% 6232 -
>> # 6 Extended offline Completed without error 00% 6170 -
>> # 7 Extended offline Completed without error 00% 6064 -
>> # 8 Extended offline Completed without error 00% 6029 -
>> # 9 Extended offline Completed without error 00% 5898 -
>> #10 Extended offline Aborted by host 60% 5893 -
>> #11 Extended offline Completed without error 00% 5728 -
>> #12 Extended offline Completed without error 00% 5706 -
>> #13 Extended offline Interrupted (host reset) 40% 5701 -
>> #14 Extended offline Interrupted (host reset) 90% 5666 -
>> #15 Extended offline Completed without error 00% 5560 -
>> #16 Extended offline Completed without error 00% 5527 -
>> #17 Extended offline Completed without error 00% 5392 -
>> #18 Extended offline Completed without error 00% 5357 -
>> #19 Extended offline Completed without error 00% 5250 -
>> #20 Extended offline Completed without error 00% 4272 -
>> #21 Extended offline Completed without error 00% 4017 -
>>
>> Note: selective self-test log revision number (0) not 1 implies that
>> no selective self-test has ever been run
>> SMART Selective self-test log data structure revision number 0
>> Note: revision number not 1 implies that no selective self-test has
>> ever been run
>> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
>> 1 0 0 Completed_read_failure [60% left] (0-65535)
>> 2 0 0 Not_testing
>> 3 0 0 Not_testing
>> 4 0 0 Not_testing
>> 5 0 0 Not_testing
>> Selective self-test flags (0x0):
>> After scanning selected spans, do NOT read-scan remainder of disk.
>> If Selective self-test is pending on power-up, resume after 0 minute delay.
>>
>> :( I do have a bad HDD. RMA is already in progress, I need to take out
>> the drive and ship it to Samsung in Holland. Will print labels at work
>> on Tuesday.
>>
>> Questions:
>>
>> * How do I remove this HDD without causing damage to the array? Is
>> this the correct way?:
>> mdadm --manage /dev/md0 --fail /dev/sdh1 # fail the device
>> mdadm --manage /dev/md0 --remove /dev/sdh1 # remove the device
>> * (shut down the system gracefully)
>> * (remove the HDD)
>> * (install new HDD)
>> * (start system)
>> sfdisk -d /dev/sde | sfdisk /dev/sdh # partition the new HDD
>> mdadm --manage /dev/md0 --add /dev/sdh1 # add the partition to the array
>>
>> * After removing the HDD, should I do another scrub?
>>
>> Thanks a lot in advance!
>>
>> /Mathias
>>
>
> I think I need to hurry up:
>
> [13957.348692] ata10.00: exception Emask 0x0 SAct 0x3 SErr 0x0 action 0x6 frozen
> [13957.348704] ata10.00: failed command: READ FPDMA QUEUED
> [13957.348716] ata10.00: cmd 60/08:00:00:ab:5a/00:00:e1:00:00/40 tag 0
> ncq 4096 in
> [13957.348719] res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask
> 0x4 (timeout)
> [13957.348724] ata10.00: status: { DRDY }
> [13957.348730] ata10.00: failed command: WRITE FPDMA QUEUED
> [13957.348741] ata10.00: cmd 61/08:08:a8:e7:8d/00:00:2e:00:00/40 tag 1
> ncq 4096 out
> [13957.348743] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask
> 0x4 (timeout)
> [13957.348749] ata10.00: status: { DRDY }
> [13957.348759] ata10: hard resetting link
> [13962.835319] ata10: link is slow to respond, please be patient (ready=0)
> [13967.368679] ata10: SRST failed (errno=-16)
> [13967.368693] ata10: hard resetting link
> [13970.988699] ata10: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> [13971.002394] ata10.00: configured for UDMA/133
> [13971.002407] ata10.00: device reported invalid CHS sector 0
> [13971.002413] ata10.00: device reported invalid CHS sector 0
> [13971.002427] ata10: EH complete
> [14001.358848] ata10.00: exception Emask 0x0 SAct 0x2 SErr 0x0 action 0x6 frozen
> [14001.358862] ata10.00: failed command: READ FPDMA QUEUED
> [14001.358887] ata10.00: cmd 60/08:08:00:ab:5a/00:00:e1:00:00/40 tag 1
> ncq 4096 in
> [14001.358890] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask
> 0x4 (timeout)
> [14001.358898] ata10.00: status: { DRDY }
> [14001.358913] ata10: hard resetting link
> [14006.845324] ata10: link is slow to respond, please be patient (ready=0)
> [14011.378656] ata10: SRST failed (errno=-16)
> [14011.378669] ata10: hard resetting link
> [14016.865323] ata10: link is slow to respond, please be patient (ready=0)
> [14021.398640] ata10: SRST failed (errno=-16)
> [14021.398652] ata10: hard resetting link
> [14026.885310] ata10: link is slow to respond, please be patient (ready=0)
> [14029.925349] ata10: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> [14029.939048] ata10.00: configured for UDMA/133
> [14029.939061] ata10.00: device reported invalid CHS sector 0
> [14029.939078] ata10: EH complete
> [14060.345358] ata10.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
> [14060.345371] ata10.00: failed command: READ FPDMA QUEUED
> [14060.345384] ata10.00: cmd 60/08:00:00:ab:5a/00:00:e1:00:00/40 tag 0
> ncq 4096 in
> [14060.345387] res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask
> 0x4 (timeout)
> [14060.345394] ata10.00: status: { DRDY }
> [14060.345407] ata10: hard resetting link
> [14065.831985] ata10: link is slow to respond, please be patient (ready=0)
> [14070.365333] ata10: SRST failed (errno=-16)
> [14070.365345] ata10: hard resetting link
> [14074.625347] ata10: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> [14074.639043] ata10.00: configured for UDMA/133
> [14074.639056] ata10.00: device reported invalid CHS sector 0
> [14074.639088] ata10: EH complete
> [14105.358687] ata10.00: NCQ disabled due to excessive errors
> [14105.358700] ata10.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
> [14105.358712] ata10.00: failed command: READ FPDMA QUEUED
> [14105.358729] ata10.00: cmd 60/08:00:00:ab:5a/00:00:e1:00:00/40 tag 0
> ncq 4096 in
> [14105.358732] res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask
> 0x4 (timeout)
> [14105.358741] ata10.00: status: { DRDY }
> [14105.358754] ata10: hard resetting link
> [14110.845314] ata10: link is slow to respond, please be patient (ready=0)
> [14115.378674] ata10: SRST failed (errno=-16)
> [14115.378689] ata10: hard resetting link
> [14119.372023] ata10: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> [14119.385704] ata10.00: configured for UDMA/133
> [14119.385716] ata10.00: device reported invalid CHS sector 0
> [14119.385743] ata10: EH complete
> [14121.527814] ata10.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6
> [14121.527824] ata10.00: edma_err_cause=00000084 pp_flags=00000001,
> dev error, EDMA self-disable
> [14121.527832] ata10.00: failed command: READ DMA EXT
> [14121.527844] ata10.00: cmd 25/00:08:00:ab:5a/00:00:e1:00:00/e0 tag 0
> dma 4096 in
> [14121.527846] res 51/89:08:00:ab:5a/89:00:e1:00:00/e0 Emask
> 0x10 (ATA bus error)
> [14121.527852] ata10.00: status: { DRDY ERR }
> [14121.527857] ata10.00: error: { ICRC }
> [14121.527867] ata10: hard resetting link
> [14127.011973] ata10: link is slow to respond, please be patient (ready=0)
> [14131.545295] ata10: SRST failed (errno=-16)
> [14131.545307] ata10: hard resetting link
> [14137.031984] ata10: link is slow to respond, please be patient (ready=0)
> [14141.565306] ata10: SRST failed (errno=-16)
> [14141.565317] ata10: hard resetting link
> [14147.051993] ata10: link is slow to respond, please be patient (ready=0)
> [14161.032035] INFO: task jbd2/dm-0-8:613 blocked for more than 120 seconds.
> [14161.032044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [14161.032050] jbd2/dm-0-8 D ffffffff81823020 0 613 2 0x00000000
> [14161.032060] ffff8800c91e11a0 0000000000000046 0000000000000000
> ffff8800cf0697c0
> [14161.032070] ffff8800cf069780 ffff8800cf069780 ffff8800c93bdfd8
> ffff8800c93bdfd8
> [14161.032079] ffff8800c91e13d8 0000000000004000 ffff880094ed19c0
> ffff8800c97e0688
> [14161.032087] Call Trace:
> [14161.032106] [<ffffffff81204c67>] ? generic_make_request+0x2f7/0x570
> [14161.032116] [<ffffffff81133200>] ? __wait_on_buffer+0x30/0x30
> [14161.032124] [<ffffffff814262e7>] ? io_schedule+0x57/0x80
> [14161.032131] [<ffffffff8113320a>] ? sleep_on_buffer+0xa/0x20
> [14161.032137] [<ffffffff81426a2f>] ? __wait_on_bit+0x4f/0x80
> [14161.032143] [<ffffffff81133200>] ? __wait_on_buffer+0x30/0x30
> [14161.032150] [<ffffffff81426add>] ? out_of_line_wait_on_bit+0x7d/0xa0
> [14161.032159] [<ffffffff81059300>] ? autoremove_wake_function+0x30/0x30
> [14161.032168] [<ffffffff811b680e>] ?
> jbd2_journal_commit_transaction+0x155e/0x16f0
> [14161.032176] [<ffffffff810592d0>] ? abort_exclusive_wait+0xb0/0xb0
> [14161.032183] [<ffffffff8142968e>] ? apic_timer_interrupt+0xe/0x20
> [14161.032191] [<ffffffff811baddd>] ? kjournald2+0xad/0x210
> [14161.032198] [<ffffffff810592d0>] ? abort_exclusive_wait+0xb0/0xb0
> [14161.032205] [<ffffffff811bad30>] ? commit_timeout+0x10/0x10
> [14161.032212] [<ffffffff81058a0f>] ? kthread+0x7f/0x90
> [14161.032219] [<ffffffff81429a54>] ? kernel_thread_helper+0x4/0x10
> [14161.032226] [<ffffffff81058990>] ? kthread_worker_fn+0x180/0x180
> [14161.032233] [<ffffffff81429a50>] ? gs_change+0xb/0xb
> [14161.032261] INFO: task squid:1659 blocked for more than 120 seconds.
> [14161.032264] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [14161.032269] squid D ffffffff81823020 0 1659 1657 0x00000000
> [14161.032277] ffff8800c9179d60 0000000000000082 ffff8800c97b0688
> ffffffff812bf074
> [14161.032285] ffff880012b818c0 ffffffff81823020 ffff8800c22c5fd8
> ffff8800c22c5fd8
> [14161.032293] ffff8800c9179f98 0000000000004000 ffff8800c22c5fd8
> 0000000000000000
> [14161.032301] Call Trace:
>
>
> :-/
>
> Currently shutting down all daemons that access the filesystem on the
> array, in attempt to umount the fs.
>
Sorry for spamming like this, it's happening in realtime. While I was
shutting down daemons it looks like MD took out the failing HDD
itself, see:
[14401.032600] INFO: task flush-253:0:25632 blocked for more than 120 seconds.
[14401.032604] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[14401.032608] flush-253:0 D ffffffff81823020 0 25632 2 0x00000000
[14401.032615] ffff88003c02bac0 0000000000000046 ffffffff81425a7f
ffff8800cf06f5c0
[14401.032623] ffff8800cf06f580 ffffffff81823020 ffff88001666ffd8
ffff88001666ffd8
[14401.032632] ffff88003c02bcf8 0000000000004000 ffff88001666ffd8
ffff88003c02bd00
[14401.032639] Call Trace:
[14401.032645] [<ffffffff81425a7f>] ? schedule+0x49f/0xcb0
[14401.032653] [<ffffffff8105f465>] ? sched_clock_local+0x15/0x80
[14401.032662] [<ffffffff81343287>] ? get_active_stripe+0x307/0x6f0
[14401.032669] [<ffffffff81032ab0>] ? try_to_wake_up+0x210/0x210
[14401.032676] [<ffffffff81346b38>] ? make_request+0x198/0x660
[14401.032683] [<ffffffff8135b4fa>] ? __map_bio+0x4a/0x1d0
[14401.032690] [<ffffffff810592d0>] ? abort_exclusive_wait+0xb0/0xb0
[14401.032697] [<ffffffff8134d563>] ? md_make_request+0x103/0x250
[14401.032704] [<ffffffff81204c67>] ? generic_make_request+0x2f7/0x570
[14401.032710] [<ffffffff810f7399>] ? kmem_cache_alloc+0x169/0x180
[14401.032718] [<ffffffff8135bff4>] ? dm_get_live_table+0x44/0x60
[14401.032724] [<ffffffff81360225>] ? linear_merge+0x45/0x50
[14401.032731] [<ffffffff81204f4d>] ? submit_bio+0x6d/0x100
[14401.032738] [<ffffffff8118296c>] ? ext4_io_submit+0x1c/0x50
[14401.032744] [<ffffffff81182ac1>] ? ext4_bio_write_page+0x121/0x370
[14401.032751] [<ffffffff8117c137>] ? mpage_da_submit_io+0x347/0x450
[14401.032759] [<ffffffff81180eae>] ? mpage_da_map_and_submit+0x1ce/0x420
[14401.032766] [<ffffffff81181920>] ? ext4_da_writepages+0x340/0x620
[14401.032774] [<ffffffff812071f7>] ? blk_flush_plug_list+0xa7/0x250
[14401.032782] [<ffffffff8112c41e>] ? writeback_single_inode+0x10e/0x270
[14401.032789] [<ffffffff8112c801>] ? writeback_sb_inodes+0xf1/0x1b0
[14401.032796] [<ffffffff814297ee>] ? reschedule_interrupt+0xe/0x20
[14401.032803] [<ffffffff8112d36b>] ? writeback_inodes_wb+0x7b/0x150
[14401.032810] [<ffffffff8112d8d3>] ? wb_writeback+0x493/0x4f0
[14401.032818] [<ffffffff8111fec2>] ? get_nr_inodes+0x42/0x60
[14401.032825] [<ffffffff8112d9c7>] ? wb_check_old_data_flush+0x97/0xa0
[14401.032832] [<ffffffff8112db3f>] ? wb_do_writeback+0x16f/0x210
[14401.032839] [<ffffffff81046dd0>] ? init_timer_deferrable_key+0x10/0x10
[14401.032846] [<ffffffff8112dc5b>] ? bdi_writeback_thread+0x7b/0x310
[14401.032852] [<ffffffff8102d149>] ? __wake_up_common+0x49/0x80
[14401.032860] [<ffffffff8112dbe0>] ? wb_do_writeback+0x210/0x210
[14401.032866] [<ffffffff81058a0f>] ? kthread+0x7f/0x90
[14401.032873] [<ffffffff81429a54>] ? kernel_thread_helper+0x4/0x10
[14401.032880] [<ffffffff81058990>] ? kthread_worker_fn+0x180/0x180
[14401.032887] [<ffffffff81429a50>] ? gs_change+0xb/0xb
[14404.915260] ata10: link is slow to respond, please be patient (ready=0)
[14409.448594] ata10: SRST failed (errno=-16)
[14409.448604] ata10: hard resetting link
[14414.935260] ata10: link is slow to respond, please be patient (ready=0)
[14431.895309] ata10: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[14431.909004] ata10.00: configured for UDMA/33
[14431.909026] ata10: EH complete
[14434.040581] ata10.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6
[14434.040589] ata10.00: edma_err_cause=00000084 pp_flags=00000001,
dev error, EDMA self-disable
[14434.040597] ata10.00: failed command: READ DMA EXT
[14434.040609] ata10.00: cmd 25/00:08:10:e0:59/00:00:b2:00:00/e0 tag 0
dma 4096 in
[14434.040612] res 51/89:08:10:e0:59/89:00:b2:00:00/e0 Emask
0x10 (ATA bus error)
[14434.040617] ata10.00: status: { DRDY ERR }
[14434.040622] ata10.00: error: { ICRC }
[14434.040631] ata10: hard resetting link
[14439.525256] ata10: link is slow to respond, please be patient (ready=0)
[14444.058589] ata10: SRST failed (errno=-16)
[14444.058600] ata10: hard resetting link
[14449.545254] ata10: link is slow to respond, please be patient (ready=0)
[14454.078594] ata10: SRST failed (errno=-16)
[14454.078604] ata10: hard resetting link
[14459.565256] ata10: link is slow to respond, please be patient (ready=0)
[14476.525289] ata10: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[14476.579007] ata10.00: configured for UDMA/33
[14476.579038] sd 9:0:0:0: [sdh] Device not ready
[14476.579043] sd 9:0:0:0: [sdh] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[14476.579049] sd 9:0:0:0: [sdh] Sense Key : Not Ready [current] [descriptor]
[14476.579057] Descriptor sense data with sense descriptors (in hex):
[14476.579061] 72 02 04 00 00 00 00 0c 00 0a 80 00 00 00 00 00
[14476.579076] b2 59 e0 10
[14476.579083] sd 9:0:0:0: [sdh] Add. Sense: Logical unit not ready,
cause not reportable
[14476.579091] sd 9:0:0:0: [sdh] CDB: Read(10): 28 00 b2 59 e0 10 00 00 08 00
[14476.579106] end_request: I/O error, dev sdh, sector 2992234512
[14476.579147] ata10: EH complete
[14484.352060] ata10.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[14484.352070] ata10.00: failed command: SMART
[14484.352082] ata10.00: cmd b0/d8:00:00:4f:c2/00:00:00:00:00/00 tag 0
[14484.352084] res 40/00:08:10:e0:59/89:00:b2:00:00/e0 Emask
0x4 (timeout)
[14484.352090] ata10.00: status: { DRDY }
[14484.352099] ata10: hard resetting link
[14489.838590] ata10: link is slow to respond, please be patient (ready=0)
[14494.371940] ata10: SRST failed (errno=-16)
[14494.371951] ata10: hard resetting link
[14499.858587] ata10: link is slow to respond, please be patient (ready=0)
[14504.391923] ata10: SRST failed (errno=-16)
[14504.391933] ata10: hard resetting link
[14509.878583] ata10: link is slow to respond, please be patient (ready=0)
[14536.995640] nfsd: last server has exited, flushing export cache
[14539.425250] ata10: SRST failed (errno=-16)
[14539.425263] ata10: hard resetting link
[14544.431917] ata10: SRST failed (errno=-16)
[14544.431926] ata10: reset failed, giving up
[14544.431932] ata10.00: disabled
[14544.431973] ata10: EH complete
[14544.432042] sd 9:0:0:0: [sdh] Unhandled error code
[14544.432062] sd 9:0:0:0: [sdh] Unhandled error code
[14544.432080] sd 9:0:0:0: [sdh] Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK
[14544.432111] sd 9:0:0:0: [sdh]
[14544.432126] sd 9:0:0:0: [sdh] CDB: Read(10)Result:
hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[14544.432163] sd 9:0:0:0: [sdh] CDB: Write(10): 2a 00 e1 5a ab 00 00 00 08
[14544.432216] sd 9:0:0:0: [sdh] Unhandled error code
[14544.432232] 00
[14544.432240] sd 9:0:0:0: [sdh]
[14544.432253] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[14544.432272] end_request: I/O error, dev sdh, sector 3780815616
[14544.432290] sd 9:0:0:0: [sdh] CDB:
[14544.432305] md/raid:md0: Disk failure on sdh1, disabling device.
[14544.432321] md/raid:md0: Operation continuing on 6 devices.
[14544.432336] Write(10): 2a 00 e1 5a ef 78 00 00 80 00
[14544.432389] end_request: I/O error, dev sdh, sector 3780833144
[14544.432412] : 28 00 b2 33 44 80 00 00 08 00
[14544.432433] end_request: I/O error, dev sdh, sector 2989704320
[14544.432453] sd 9:0:0:0: [sdh] Unhandled error code
[14544.432471] sd 9:0:0:0: [sdh] Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK
[14544.432498] sd 9:0:0:0: [sdh] CDB: Write(10): 2a 00 e1 5a ef f8 00 00 80 00
[14544.432553] end_request: I/O error, dev sdh, sector 3780833272
[14544.432618] sd 9:0:0:0: [sdh] Unhandled error code
[14544.432639] sd 9:0:0:0: [sdh] Unhandled error code
[14544.432659] sd 9:0:0:0: [sdh] Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK
[14544.432687] sd 9:0:0:0: [sdh] CDB: Write(10): 2a 00 b2 59 e0 10 00 00 08 00
[14544.432749] end_request: I/O error, dev sdh, sector 2992234512
[14544.432774] sd 9:0:0:0: [sdh] Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK
[14544.432785] sd 9:0:0:0: [sdh] CDB: Write(10): 2a 00
[14544.432800] sd 9:0:0:0: [sdh] Unhandled error code
[14544.432820] sd 9:0:0:0: [sdh] Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK
[14544.432849] sd 9:0:0:0: [sdh] CDB: Write(10): 2a 00 e1 5a eb 00 00 00 80 00
[14544.432910] end_request: I/O error, dev sdh, sector 3780832000
[14544.432933] e1 5a f0 78 00 00 80 00
[14544.432949] end_request: I/O error, dev sdh, sector 3780833400
[14544.432965] sd 9:0:0:0: [sdh] Unhandled error code
[14544.432972] sd 9:0:0:0: [sdh] Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK
[14544.432982] sd 9:0:0:0: [sdh] CDB: Write(10): 2a 00 e1 5a ef 00 00 00 78 00
[14544.433002] end_request: I/O error, dev sdh, sector 3780833024
[14544.433012] sd 9:0:0:0: [sdh] Unhandled error code
[14544.433033] sd 9:0:0:0: [sdh] Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK
[14544.433062] sd 9:0:0:0: [sdh] CDB: Write(10): 2a 00 e1 5a f0 f8 00 00 80 00
[14544.433121] end_request: I/O error, dev sdh, sector 3780833528
[14544.433205] sd 9:0:0:0: [sdh] Unhandled error code
[14544.433227] sd 9:0:0:0: [sdh] Unhandled error code
[14544.433247] sd 9:0:0:0: [sdh] Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK
[14544.433276] sd 9:0:0:0: [sdh] CDB: Write(10): 2a 00 e1 5a f1 78 00 00 80 00
[14544.433333] end_request: I/O error, dev sdh, sector 3780833656
[14544.433357] sd 9:0:0:0: [sdh] Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK
[14544.433366] sd 9:0:0:0: [sdh] CDB: Write(10): 2a 00 2e 8d e7 b0 00 00 28 00
[14544.433387] end_request: I/O error, dev sdh, sector 781051824
[14544.433399] sd 9:0:0:0: [sdh] Unhandled error code
[14544.433408] sd 9:0:0:0: [sdh] Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK
[14544.433419] sd 9:0:0:0: [sdh] CDB: Write(10): 2a 00 e1 5a f1 f8 00 00 80 00
[14544.433440] end_request: I/O error, dev sdh, sector 3780833784
[14544.433487] sd 9:0:0:0: [sdh] Unhandled error code
[14544.433493] sd 9:0:0:0: [sdh] Result: hostbyte=DID_BAD_TARGET
driverbyte=DRIVER_OK
[14544.433502] sd 9:0:0:0: [sdh] CDB: Write(10): 2a 00 e1 5a f2 78 00 00 08 00
[14544.433523] end_request: I/O error, dev sdh, sector 3780833912
[14544.488761] RAID conf printout:
[14544.488772] --- level:6 rd:7 wd:6
[14544.488779] disk 0, o:1, dev:sdg1
[14544.488783] disk 1, o:1, dev:sdb1
[14544.488788] disk 2, o:1, dev:sdd1
[14544.488793] disk 3, o:1, dev:sdc1
[14544.488797] disk 4, o:1, dev:sdf1
[14544.488801] disk 5, o:0, dev:sdh1
[14544.488805] disk 6, o:1, dev:sde1
[14544.515279] RAID conf printout:
[14544.515289] --- level:6 rd:7 wd:6
[14544.515295] disk 0, o:1, dev:sdg1
[14544.515299] disk 1, o:1, dev:sdb1
[14544.515304] disk 2, o:1, dev:sdd1
[14544.515308] disk 3, o:1, dev:sdc1
[14544.515313] disk 4, o:1, dev:sdf1
[14544.515317] disk 6, o:1, dev:sde1
[14570.639011] nvidia 0000:00:03.5: PCI INT B disabled
[14570.639441] nvidia 0000:03:00.0: PCI INT A disabled
[14591.505699] HDA Intel 0000:00:08.0: PCI INT A disabled
~ $ dmesg > /srv/http/dmesg-failing-harddrive.log
~ $ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdh1[6](F) sdg1[0] sdf1[5] sde1[7] sdd1[4] sdb1[1] sdc1[3]
9751756800 blocks super 1.2 level 6, 64k chunk, algorithm 2
[7/6] [UUUUU_U]
unused devices: <none>
~ $
$ mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Tue Oct 19 08:58:41 2010
Raid Level : raid6
Array Size : 9751756800 (9300.00 GiB 9985.80 GB)
Used Dev Size : 1950351360 (1860.00 GiB 1997.16 GB)
Raid Devices : 7
Total Devices : 7
Persistence : Superblock is persistent
Update Time : Sun Jul 31 19:07:25 2011
State : clean, degraded
Active Devices : 6
Working Devices : 6
Failed Devices : 1
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : ion:0 (local to host ion)
UUID : e6595c64:b3ae90b3:f01133ac:3f402d20
Events : 6158774
Number Major Minor RaidDevice State
0 8 97 0 active sync /dev/sdg1
1 8 17 1 active sync /dev/sdb1
4 8 49 2 active sync /dev/sdd1
3 8 33 3 active sync /dev/sdc1
5 8 81 4 active sync /dev/sdf1
5 0 0 5 removed
7 8 65 6 active sync /dev/sde1
6 8 113 - faulty spare /dev/sdh1
Shutting down the LV:
$ vgchange -an lvstorage
/dev/sdh: read failed after 0 of 4096 at 0: Input/output error
/dev/sdh: read failed after 0 of 4096 at 2000398843904: Input/output error
/dev/sdh: read failed after 0 of 4096 at 2000398925824: Input/output error
/dev/sdh: read failed after 0 of 4096 at 4096: Input/output error
/dev/sdh1: read failed after 0 of 4096 at 2000397795328: Input/output error
/dev/sdh1: read failed after 0 of 4096 at 2000397877248: Input/output error
/dev/sdh1: read failed after 0 of 4096 at 0: Input/output error
/dev/sdh1: read failed after 0 of 4096 at 4096: Input/output error
0 logical volume(s) in volume group "lvstorage" now active
Removing the bad HDD:
$ mdadm --manage /dev/md0 --remove /dev/sdh1
mdadm: hot removed /dev/sdh1 from /dev/md0
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdg1[0] sdf1[5] sde1[7] sdd1[4] sdb1[1] sdc1[3]
9751756800 blocks super 1.2 level 6, 64k chunk, algorithm 2
[7/6] [UUUUU_U]
unused devices: <none>
$ mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Tue Oct 19 08:58:41 2010
Raid Level : raid6
Array Size : 9751756800 (9300.00 GiB 9985.80 GB)
Used Dev Size : 1950351360 (1860.00 GiB 1997.16 GB)
Raid Devices : 7
Total Devices : 6
Persistence : Superblock is persistent
Update Time : Sun Jul 31 19:13:02 2011
State : clean, degraded
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : ion:0 (local to host ion)
UUID : e6595c64:b3ae90b3:f01133ac:3f402d20
Events : 6158779
Number Major Minor RaidDevice State
0 8 97 0 active sync /dev/sdg1
1 8 17 1 active sync /dev/sdb1
4 8 49 2 active sync /dev/sdd1
3 8 33 3 active sync /dev/sdc1
5 8 81 4 active sync /dev/sdf1
5 0 0 5 removed
7 8 65 6 active sync /dev/sde1
I'll shut down the system now and remove the HDD. I suppose I've just
one question to ask then; should I rescrub the array when it's up with
1 HDD removed?
Thanks again,
Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Do I have a bad HDD?
2011-07-31 18:14 ` Mathias Burén
@ 2011-07-31 18:25 ` Johannes Truschnigg
2011-07-31 18:31 ` Mathias Burén
0 siblings, 1 reply; 9+ messages in thread
From: Johannes Truschnigg @ 2011-07-31 18:25 UTC (permalink / raw)
To: Mathias Burén; +Cc: Linux-RAID
[-- Attachment #1: Type: text/plain, Size: 1859 bytes --]
Hi Mathias,
PLEASE don't always quote everything you've just written a few minutes
ago when replying to yourself. This makes these responses really, really
inconvenient to read.
Concerning your problem at hand: Don't panic, everything should be fine.
Your RAID did what it is supposed to do: protect your from a single
drive failure without inducing downtime. A disk has failed, and now it's
time to replace it, that's nothing out of the ordinary. You don't need
to shut down your system - RAID systems have "no service interruption in
case of an accident" as a design goal, and md is rather good at meeting
just that.
If your SATA controller supports hotplug (and if it's SATA-300, that
much is for certain), just unplug the old drive, replace it with a new
one, clone your partition table setup (if any) from an old driver to the
new one, have md pick up the disk and integrate it into your array, and
see how everything'll be taken care of automatically. md will resync the
array once the new disk is part of the array, and it will be smooth
sailing again afterwards. Just make sure you don't pull out the wrong
drive, but correctly identify the broken one. ;)
While the array's being worked on, you can do whatever you intended to
do with your seemingly healthy array from of a few hours ago - the only
difference it that (some) things will go slower, but that's about it.
The only serious problem you could run into is a second and third
harddrive failing while your array isn't 100% OK again yet - but even in
that case, you'd have a backup ready, now wouldn't you? :)
--
with best regards:
- Johannes Truschnigg ( johannes@truschnigg.info )
www: http://johannes.truschnigg.info/
phone: +43 650 2 133337
xmpp: johannes@truschnigg.info
Please do not bother me with HTML-eMail or attachments. Thank you.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Do I have a bad HDD?
2011-07-31 18:25 ` Johannes Truschnigg
@ 2011-07-31 18:31 ` Mathias Burén
2011-07-31 23:42 ` Phil Turmel
0 siblings, 1 reply; 9+ messages in thread
From: Mathias Burén @ 2011-07-31 18:31 UTC (permalink / raw)
To: Johannes Truschnigg; +Cc: Linux-RAID
On 31 July 2011 19:25, Johannes Truschnigg <johannes@truschnigg.info> wrote:
> Hi Mathias,
>
> PLEASE don't always quote everything you've just written a few minutes
> ago when replying to yourself. This makes these responses really, really
> inconvenient to read.
>
> Concerning your problem at hand: Don't panic, everything should be fine.
> Your RAID did what it is supposed to do: protect your from a single
> drive failure without inducing downtime. A disk has failed, and now it's
> time to replace it, that's nothing out of the ordinary. You don't need
> to shut down your system - RAID systems have "no service interruption in
> case of an accident" as a design goal, and md is rather good at meeting
> just that.
>
> If your SATA controller supports hotplug (and if it's SATA-300, that
> much is for certain), just unplug the old drive, replace it with a new
> one, clone your partition table setup (if any) from an old driver to the
> new one, have md pick up the disk and integrate it into your array, and
> see how everything'll be taken care of automatically. md will resync the
> array once the new disk is part of the array, and it will be smooth
> sailing again afterwards. Just make sure you don't pull out the wrong
> drive, but correctly identify the broken one. ;)
>
> While the array's being worked on, you can do whatever you intended to
> do with your seemingly healthy array from of a few hours ago - the only
> difference it that (some) things will go slower, but that's about it.
> The only serious problem you could run into is a second and third
> harddrive failing while your array isn't 100% OK again yet - but even in
> that case, you'd have a backup ready, now wouldn't you? :)
>
>
> --
> with best regards:
> - Johannes Truschnigg ( johannes@truschnigg.info )
>
> www: http://johannes.truschnigg.info/
> phone: +43 650 2 133337
> xmpp: johannes@truschnigg.info
>
> Please do not bother me with HTML-eMail or attachments. Thank you.
>
>
Sorry about that!
No I don't have a backup ;), perhaps it's time to get one. This isn't
super important data, more like "nice to have".
I don't have any hotplug bays or anything like that (you'll see in a
while after I put up the pictures) so I shut down the system to take
the faulty HDD out.
Is it wise to run a check on the array while it's degraded by 1 HDD?
Thanks again,
/Mathias
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Do I have a bad HDD?
2011-07-31 18:31 ` Mathias Burén
@ 2011-07-31 23:42 ` Phil Turmel
2011-07-31 23:56 ` Mathias Burén
0 siblings, 1 reply; 9+ messages in thread
From: Phil Turmel @ 2011-07-31 23:42 UTC (permalink / raw)
To: Mathias Burén; +Cc: Johannes Truschnigg, Linux-RAID
Good evening Mathias,
On 07/31/2011 02:31 PM, Mathias Burén wrote:
[trim]
> No I don't have a backup ;), perhaps it's time to get one. This isn't
> super important data, more like "nice to have".
>
> I don't have any hotplug bays or anything like that (you'll see in a
> while after I put up the pictures) so I shut down the system to take
> the faulty HDD out.
>
> Is it wise to run a check on the array while it's degraded by 1 HDD?
It shouldn't hurt your array, other than a bit of exercise. If another
drive is marginal, it could conceivably "push it over the edge".
Any positive value it has will occur anyways during the resync with the
replacement drive for sdh.
I wouldn't bother running the scrub until the resync is finished.
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Do I have a bad HDD?
2011-07-31 23:42 ` Phil Turmel
@ 2011-07-31 23:56 ` Mathias Burén
2011-08-22 17:10 ` Mathias Burén
0 siblings, 1 reply; 9+ messages in thread
From: Mathias Burén @ 2011-07-31 23:56 UTC (permalink / raw)
To: Phil Turmel; +Cc: Johannes Truschnigg, Linux-RAID
On 1 August 2011 00:42, Phil Turmel <philip@turmel.org> wrote:
> Good evening Mathias,
>
> On 07/31/2011 02:31 PM, Mathias Burén wrote:
>
> [trim]
>
>> No I don't have a backup ;), perhaps it's time to get one. This isn't
>> super important data, more like "nice to have".
>>
>> I don't have any hotplug bays or anything like that (you'll see in a
>> while after I put up the pictures) so I shut down the system to take
>> the faulty HDD out.
>>
>> Is it wise to run a check on the array while it's degraded by 1 HDD?
>
> It shouldn't hurt your array, other than a bit of exercise. If another
> drive is marginal, it could conceivably "push it over the edge".
>
> Any positive value it has will occur anyways during the resync with the
> replacement drive for sdh.
>
> I wouldn't bother running the scrub until the resync is finished.
>
> Phil
>
Thanks, I'll wait to scrub until I get the new HDD (hopefully within 2
weeks). Pics of the beast here: http://stuff3.imgur.com/htpcnas
/Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Do I have a bad HDD?
2011-07-31 23:56 ` Mathias Burén
@ 2011-08-22 17:10 ` Mathias Burén
0 siblings, 0 replies; 9+ messages in thread
From: Mathias Burén @ 2011-08-22 17:10 UTC (permalink / raw)
Cc: Linux-RAID
On 1 August 2011 00:56, Mathias Burén <mathias.buren@gmail.com> wrote:
> On 1 August 2011 00:42, Phil Turmel <philip@turmel.org> wrote:
>> Good evening Mathias,
>>
>> On 07/31/2011 02:31 PM, Mathias Burén wrote:
>>
>> [trim]
>>
>>> No I don't have a backup ;), perhaps it's time to get one. This isn't
>>> super important data, more like "nice to have".
>>>
>>> I don't have any hotplug bays or anything like that (you'll see in a
>>> while after I put up the pictures) so I shut down the system to take
>>> the faulty HDD out.
>>>
>>> Is it wise to run a check on the array while it's degraded by 1 HDD?
>>
>> It shouldn't hurt your array, other than a bit of exercise. If another
>> drive is marginal, it could conceivably "push it over the edge".
>>
>> Any positive value it has will occur anyways during the resync with the
>> replacement drive for sdh.
>>
>> I wouldn't bother running the scrub until the resync is finished.
>>
>> Phil
>>
>
>
> Thanks, I'll wait to scrub until I get the new HDD (hopefully within 2
> weeks). Pics of the beast here: http://stuff3.imgur.com/htpcnas
>
> /Mathias
>
New HDD received from Samsung via RMA, all went fine adding it, now
the array is in recovery:
root@ion ~ $ mdadm --manage /dev/md0 --add /dev/sdh1
mdadm: added /dev/sdh1
root@ion ~ $ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid6 sdh1[8] sdf1[5] sde1[7] sdg1[0] sdd1[4] sdb1[1] sdc1[3]
9751756800 blocks super 1.2 level 6, 64k chunk, algorithm 2
[7/6] [UUUUU_U]
[>....................] recovery = 0.0% (87852/1950351360)
finish=1109.9min speed=29284K/sec
unused devices: <none>
Thanks for all the help.
/Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-08-22 17:10 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-07-31 13:05 Do I have a bad HDD? Mathias Burén
2011-07-31 17:59 ` Mathias Burén
2011-07-31 18:05 ` Mathias Burén
2011-07-31 18:14 ` Mathias Burén
2011-07-31 18:25 ` Johannes Truschnigg
2011-07-31 18:31 ` Mathias Burén
2011-07-31 23:42 ` Phil Turmel
2011-07-31 23:56 ` Mathias Burén
2011-08-22 17:10 ` Mathias Burén
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox