* Re: All drive in Raid 5 are in 'spare' mode
From: Phil Turmel @ 2015-02-16 20:03 UTC (permalink / raw)
To: Dush; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <CAL7hTOfhWiCmKVVCtpmHuB-OJkZjQsuaKZMN8oFKAArh=NuSHQ@mail.gmail.com>
Hi Dush,
{Convention on kernel.org is to trim replies and either bottom post or
interleave. Please don't top-post.}
On 02/16/2015 02:38 PM, Dush wrote:
> Hi Phil,
>
> Thanks for your answer!
>
> Unfortunately, I think I just loosed a disk (sde)... I don't see it
> anymore in /dev , I have in dmesg:
> d
> [ 12.280021] ata7: softreset failed (1st FIS failed)
> [ 22.280019] ata7: softreset failed (1st FIS failed)
> [ 57.280015] ata7: softreset failed (1st FIS failed)
> [ 57.280222] ata7: limiting SATA link speed to 1.5 Gbps
> [ 62.453345] ata7: softreset failed (device not ready)
> [ 62.453558] ata7: reset failed, giving up
Where are the forensics I asked for as "Step one"? Did you read about
and fix any timeout mismatch issue?
[trim /]
> You was right, I already tried to start the raid and it succeed to do
> it with 3 drives: b, c and e. Then I added the d because I thought it
> was de-synchronized.
> Now I think my drive e was out of this raid for a while and I started
> to had trouble because d started to had some issues.
>
> Is it possible to force raid to start with b, c and d (forcing d to be
> 'normal')? Time for me to copy everything to another drive...
No. sdd3 was converted to a spare.
Also, your device names have changed. You *must* keep track of which
one is which "RaidDevice". Specifically, what was "sde" now appears to
be "sdd". Did you reboot? You know that device names are not
guaranteed to be consistent from one boot to the next, I hope. Show an
excerpt from "ls -l /dev/disk/by-id/" with your next report so we know
which drive serial number has which name.
Phil
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Jes Sorensen @ 2015-02-16 20:36 UTC (permalink / raw)
To: NeilBrown; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <20150203093040.569aa5e1@notabene.brown>
NeilBrown <neilb@suse.de> writes:
> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
> wrote:
>
>> Dear All,
>> Any updates on this issue.
>
> Probably the same as:
>
> http://marc.info/?l=linux-raid&m=142283560704091&w=2
Hi Neil,
I ran some tests on this one against the latest Linus' tree as of today
(1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
your pending 3.20 patches.
I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
run fio on top of a device while it is resyncing and I fail one of the
devices.
I can reproduce the issue for raid4 and raid5, but I don't see it if I
I use a raid6.
The following sequence consistently reproduces the problem for me:
mdadm -C /dev/md111 -f -e 1.2 -l5 -n4 /dev/sd[ghij]3
fio --name=md111 --filename=/dev/md111 --thread --numjobs=10 --direct=1 --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1 --norandommap --scramble_buffers=1 --stonewall --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000 --blockalign=512
mdadm /dev/md111 -f /dev/sdg3
Cheers,
Jes
^ permalink raw reply
* Re: All drive in Raid 5 are in 'spare' mode
From: Dush @ 2015-02-16 22:24 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <54E24D10.7030901@turmel.org>
> {Convention on kernel.org is to trim replies and either bottom post or
> interleave. Please don't top-post.}
Sorry about that...
> Where are the forensics I asked for as "Step one"?
# dmesg
[ 12.273354] ata7: softreset failed (1st FIS failed)
[ 22.273352] ata7: softreset failed (1st FIS failed)
[ 57.273349] ata7: softreset failed (1st FIS failed)
[ 57.273556] ata7: limiting SATA link speed to 1.5 Gbps
[ 62.446679] ata7: softreset failed (device not ready)
[ 62.446892] ata7: reset failed, giving up
[ 62.933371] ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 62.939016] ata8.00: ATA-8: SAMSUNG HD103SJ, 1AJ10001, max UDMA/133
[ 62.939019] ata8.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[ 62.944696] ata8.00: configured for UDMA/133
[ 0.380968] random: systemd-tmpfile urandom read with 1 bits of
entropy available
[ 0.411310] ata1: PATA max UDMA/100 cmd 0xdc00 ctl 0xd880 bmdma 0xd400 irq 17
[ 0.411312] ata2: PATA max UDMA/100 cmd 0xd800 ctl 0xd480 bmdma 0xd408 irq 17
[ 63.473837] systemd[1]: Cannot add dependency job for unit
gssproxy.service, ignoring: Unit gssproxy.service failed to load: No
such file or directory.
[ 63.632259] systemd-journald[157]: Received request to flush
runtime journal from PID 1
[ 63.884029] md: bind<sdc2>
[ 63.895136] md: bind<sdc3>
[ 63.910832] md: bind<sdb2>
[ 63.926903] md: bind<sdd3>
[ 63.937151] md: bind<sdd2>
[ 63.940172] md: bind<sdb3>
[ 94.538293] md: raid6 personality registered for level 6
[ 94.538298] md: raid5 personality registered for level 5
[ 94.538300] md: raid4 personality registered for level 4
[ 94.538954] md/raid:md127: device sdd2 operational as raid disk 1
[ 94.538958] md/raid:md127: device sdb2 operational as raid disk 3
[ 94.538960] md/raid:md127: device sdc2 operational as raid disk 0
[ 94.539457] md/raid:md127: allocated 0kB
[ 94.539493] md/raid:md127: raid level 5 active with 3 out of 4
devices, algorithm 2
[ 94.539568] md127: detected capacity change from 0 to 16927358976
[ 94.544725] md127: unknown partition table
# smartctl -x /dev/sdb
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.18.6-1-ARCH] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.11
Device Model: ST3500320AS
Serial Number: 9QM8H48M
LU WWN Device Id: 5 000c50 01099d533
Firmware Version: SD15
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 2.6, 3.0 Gb/s
Local Time is: Mon Feb 16 22:09:04 2015 GMT
==> WARNING: There are known problems with these drives,
THIS DRIVE MAY OR MAY NOT BE AFFECTED,
see the following web pages for details:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/207951en
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=632758
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Unknown
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 650) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 120) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x103b) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 099 088 006 - 160388856
3 Spin_Up_Time PO---- 094 088 000 - 0
4 Start_Stop_Count -O--CK 099 099 020 - 1288
5 Reallocated_Sector_Ct PO--CK 100 100 036 - 1290
7 Seek_Error_Rate POSR-- 069 060 030 - 77467799408
9 Power_On_Hours -O--CK 061 061 000 - 34392
10 Spin_Retry_Count PO--C- 100 100 097 - 0
12 Power_Cycle_Count -O--CK 100 100 020 - 988
184 End-to-End_Error -O--CK 100 100 099 - 0
187 Reported_Uncorrect -O--CK 001 001 000 - 183
188 Command_Timeout -O--CK 100 096 000 - 4295032839
189 High_Fly_Writes -O-RCK 100 100 000 - 0
190 Airflow_Temperature_Cel -O---K 067 054 045 - 33 (Min/Max 16/33)
194 Temperature_Celsius -O---K 033 046 000 - 33 (0 10 0 0 0)
195 Hardware_ECC_Recovered -O-RC- 024 016 000 - 160388856
197 Current_Pending_Sector -O--C- 100 100 000 - 757
198 Offline_Uncorrectable ----C- 100 100 000 - 757
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 GPL,SL R/O 1 Summary SMART error log
0x02 GPL,SL R/O 5 Comprehensive SMART error log
0x03 GPL,SL R/O 5 Ext. Comprehensive SMART error log
0x06 GPL,SL R/O 1 SMART self-test log
0x07 GPL,SL R/O 1 Extended self-test log
0x09 GPL,SL R/W 1 Selective self-test log
0x10 GPL,SL R/O 1 NCQ Command Error log
0x11 GPL,SL R/O 1 SATA Phy Event Counters
0x21 GPL,SL R/O 1 Write stream error log
0x22 GPL,SL R/O 1 Read stream error log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa1 GPL,SL VS 20 Device vendor specific log
0xa2 GPL VS 2248 Device vendor specific log
0xa8 GPL,SL VS 20 Device vendor specific log
0xa9 GPL,SL VS 1 Device vendor specific log
0xb0 GPL VS 2819 Device vendor specific log
0xbe-0xbf GPL VS 65535 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
Device Error Count: 261 (device log contains only the most recent 20 errors)
CR = Command Register
FEATR = Features Register
COUNT = Count (was: Sector Count) Register
LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8
LH = LBA High (was: Cylinder High) Register ] LBA
LM = LBA Mid (was: Cylinder Low) Register ] Register
LL = LBA Low (was: Sector Number) Register ]
DV = Device (was: Device/Head) Register
DC = Device Control Register
ER = Error register
ST = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 261 [0] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d e0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
a1 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:14.067 IDENTIFY PACKET DEVICE
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:14.067 IDENTIFY DEVICE
00 00 00 00 00 00 00 00 00 00 00 00 04 13:02:13.913 NOP [Abort
queued commands]
00 00 00 00 00 00 00 00 00 00 00 00 ff 13:02:13.587 NOP [Abort
queued commands]
a1 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.581 IDENTIFY PACKET DEVICE
Error 260 [19] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d e0 00
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:14.067 IDENTIFY DEVICE
00 00 00 00 00 00 00 00 00 00 00 00 04 13:02:13.913 NOP [Abort
queued commands]
00 00 00 00 00 00 00 00 00 00 00 00 ff 13:02:13.587 NOP [Abort
queued commands]
a1 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.581 IDENTIFY PACKET DEVICE
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.580 IDENTIFY DEVICE
Error 259 [18] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d e0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
a1 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.581 IDENTIFY PACKET DEVICE
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.580 IDENTIFY DEVICE
00 00 00 00 00 00 00 00 00 00 00 00 04 13:02:08.427 NOP [Abort
queued commands]
00 00 00 00 00 00 00 00 00 00 00 00 ff 13:02:08.107 NOP [Abort
queued commands]
a1 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.086 IDENTIFY PACKET DEVICE
Error 258 [17] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d e0 00
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.580 IDENTIFY DEVICE
00 00 00 00 00 00 00 00 00 00 00 00 04 13:02:08.427 NOP [Abort
queued commands]
00 00 00 00 00 00 00 00 00 00 00 00 ff 13:02:08.107 NOP [Abort
queued commands]
a1 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.086 IDENTIFY PACKET DEVICE
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.069 IDENTIFY DEVICE
Error 257 [16] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d e0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
a1 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.086 IDENTIFY PACKET DEVICE
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.069 IDENTIFY DEVICE
2f 00 00 00 01 00 00 00 00 00 10 a0 00 13:02:08.047 READ LOG EXT
60 00 00 00 10 00 00 81 00 05 ef 40 00 13:02:04.913 READ FPDMA QUEUED
60 00 00 00 08 00 00 83 00 04 9f 40 00 13:02:04.913 READ FPDMA QUEUED
Error 256 [15] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d e0 00
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.069 IDENTIFY DEVICE
2f 00 00 00 01 00 00 00 00 00 10 a0 00 13:02:08.047 READ LOG EXT
60 00 00 00 10 00 00 81 00 05 ef 40 00 13:02:04.913 READ FPDMA QUEUED
60 00 00 00 08 00 00 83 00 04 9f 40 00 13:02:04.913 READ FPDMA QUEUED
60 00 00 00 08 00 00 41 00 08 47 40 00 13:02:04.912 READ FPDMA QUEUED
Error 255 [14] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d e0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
2f 00 00 00 01 00 00 00 00 00 10 a0 00 13:02:08.047 READ LOG EXT
60 00 00 00 10 00 00 81 00 05 ef 40 00 13:02:04.913 READ FPDMA QUEUED
60 00 00 00 08 00 00 83 00 04 9f 40 00 13:02:04.913 READ FPDMA QUEUED
60 00 00 00 08 00 00 41 00 08 47 40 00 13:02:04.912 READ FPDMA QUEUED
60 00 00 00 08 00 00 70 00 06 b7 40 00 13:02:04.909 READ FPDMA QUEUED
Error 254 [13] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d 40 00 Device Fault; Error: ABRT at
LBA = 0x0000329d = 12957
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 00 00 10 00 00 81 00 05 ef 40 00 13:02:04.913 READ FPDMA QUEUED
60 00 00 00 08 00 00 83 00 04 9f 40 00 13:02:04.913 READ FPDMA QUEUED
60 00 00 00 08 00 00 41 00 08 47 40 00 13:02:04.912 READ FPDMA QUEUED
60 00 00 00 08 00 00 70 00 06 b7 40 00 13:02:04.909 READ FPDMA QUEUED
60 00 00 00 08 00 00 7e 00 03 87 40 00 13:02:04.909 READ FPDMA QUEUED
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 34220
336941
# 2 Extended offline Completed: read failure 90% 32539
39315653
# 3 Extended offline Completed without error 00% 25079 -
# 4 Extended offline Interrupted (host reset) 00% 24385 -
# 5 Short offline Completed without error 00% 19163 -
# 6 Short offline Completed without error 00% 10742 -
# 7 Extended offline Completed without error 00% 79 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 522 (0x020a)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 33 Celsius
Power Cycle Min/Max Temperature: 16/33 Celsius
Lifetime Min/Max Temperature: 10/46 Celsius
Under/Over Temperature Limit Count: 0/235
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/ 0 Celsius
Min/Max Temperature Limit: 0/ 0 Celsius
Temperature History Size (Index): 128 (50)
Index Estimated Time Temperature Celsius
51 2015-02-16 20:02 30 ***********
... ..( 12 skipped). .. ***********
64 2015-02-16 20:15 30 ***********
65 2015-02-16 20:16 28 *********
66 2015-02-16 20:17 28 *********
67 2015-02-16 20:18 28 *********
68 2015-02-16 20:19 29 **********
... ..( 6 skipped). .. **********
75 2015-02-16 20:26 29 **********
76 2015-02-16 20:27 30 ***********
... ..( 9 skipped). .. ***********
86 2015-02-16 20:37 30 ***********
87 2015-02-16 20:38 31 ************
... ..( 40 skipped). .. ************
0 2015-02-16 21:19 31 ************
1 2015-02-16 21:20 32 *************
... ..( 38 skipped). .. *************
40 2015-02-16 21:59 32 *************
41 2015-02-16 22:00 33 **************
... ..( 8 skipped). .. **************
50 2015-02-16 22:09 33 **************
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
Device Statistics (GP Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x000a 2 14 Device-to-host register FISes sent due to a COMRESET
0x0001 2 0 Command failed due to ICRC error
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
# smartctl -x /dev/sdc
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.18.6-1-ARCH] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Black
Device Model: WDC WD5001AALS-00E3A0
Serial Number: WD-WCATR3121296
LU WWN Device Id: 5 0014ee 25a457f4e
Firmware Version: 05.01D05
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.6, 3.0 Gb/s
Local Time is: Mon Feb 16 22:09:29 2015 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Disabled
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 8700) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 103) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x3037) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 179 174 021 - 4050
4 Start_Stop_Count -O--CK 100 100 000 - 963
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 068 068 000 - 23596
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 739
192 Power-Off_Retract_Count -O--CK 200 200 000 - 97
193 Load_Cycle_Count -O--CK 200 200 000 - 865
194 Temperature_Celsius -O---K 111 097 000 - 36
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 200 200 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb5 GPL,SL VS 1 Device vendor specific log
0xb6 GPL VS 1 Device vendor specific log
0xb7 GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 24 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 23424 -
# 2 Extended offline Completed without error 00% 21741 -
# 3 Extended offline Completed without error 00% 14306 -
# 4 Short offline Completed without error 00% 8409 -
# 5 Short offline Completed without error 00% 2 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 2
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 36 Celsius
Power Cycle Min/Max Temperature: 18/37 Celsius
Lifetime Min/Max Temperature: 36/50 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (252)
Index Estimated Time Temperature Celsius
253 2015-02-16 14:12 35 ****************
... ..( 6 skipped). .. ****************
260 2015-02-16 14:19 35 ****************
261 2015-02-16 14:20 36 *****************
... ..( 16 skipped). .. *****************
278 2015-02-16 14:37 36 *****************
279 2015-02-16 14:38 37 ******************
280 2015-02-16 14:39 36 *****************
... ..( 2 skipped). .. *****************
283 2015-02-16 14:42 36 *****************
284 2015-02-16 14:43 37 ******************
285 2015-02-16 14:44 36 *****************
... ..( 54 skipped). .. *****************
340 2015-02-16 15:39 36 *****************
341 2015-02-16 15:40 37 ******************
342 2015-02-16 15:41 36 *****************
... ..( 3 skipped). .. *****************
346 2015-02-16 15:45 36 *****************
347 2015-02-16 15:46 37 ******************
348 2015-02-16 15:47 36 *****************
... ..( 4 skipped). .. *****************
353 2015-02-16 15:52 36 *****************
354 2015-02-16 15:53 37 ******************
355 2015-02-16 15:54 36 *****************
... ..( 2 skipped). .. *****************
358 2015-02-16 15:57 36 *****************
359 2015-02-16 15:58 37 ******************
360 2015-02-16 15:59 36 *****************
... ..( 10 skipped). .. *****************
371 2015-02-16 16:10 36 *****************
372 2015-02-16 16:11 37 ******************
373 2015-02-16 16:12 36 *****************
... ..( 12 skipped). .. *****************
386 2015-02-16 16:25 36 *****************
387 2015-02-16 16:26 37 ******************
388 2015-02-16 16:27 36 *****************
... ..(145 skipped). .. *****************
56 2015-02-16 18:53 36 *****************
57 2015-02-16 18:54 35 ****************
... ..( 46 skipped). .. ****************
104 2015-02-16 19:41 35 ****************
105 2015-02-16 19:42 ? -
106 2015-02-16 19:43 18 -
107 2015-02-16 19:44 19 -
108 2015-02-16 19:45 20 *
109 2015-02-16 19:46 20 *
110 2015-02-16 19:47 21 **
111 2015-02-16 19:48 21 **
112 2015-02-16 19:49 22 ***
113 2015-02-16 19:50 22 ***
114 2015-02-16 19:51 23 ****
115 2015-02-16 19:52 24 *****
116 2015-02-16 19:53 25 ******
117 2015-02-16 19:54 25 ******
118 2015-02-16 19:55 25 ******
119 2015-02-16 19:56 26 *******
... ..( 3 skipped). .. *******
123 2015-02-16 20:00 26 *******
124 2015-02-16 20:01 27 ********
125 2015-02-16 20:02 27 ********
126 2015-02-16 20:03 27 ********
127 2015-02-16 20:04 28 *********
128 2015-02-16 20:05 28 *********
129 2015-02-16 20:06 29 **********
... ..( 4 skipped). .. **********
134 2015-02-16 20:11 29 **********
135 2015-02-16 20:12 30 ***********
... ..( 4 skipped). .. ***********
140 2015-02-16 20:17 30 ***********
141 2015-02-16 20:18 31 ************
142 2015-02-16 20:19 31 ************
143 2015-02-16 20:20 30 ***********
144 2015-02-16 20:21 31 ************
... ..( 3 skipped). .. ************
148 2015-02-16 20:25 31 ************
149 2015-02-16 20:26 32 *************
... ..( 18 skipped). .. *************
168 2015-02-16 20:45 32 *************
169 2015-02-16 20:46 33 **************
... ..( 16 skipped). .. **************
186 2015-02-16 21:03 33 **************
187 2015-02-16 21:04 34 ***************
... ..( 2 skipped). .. ***************
190 2015-02-16 21:07 34 ***************
191 2015-02-16 21:08 35 ****************
192 2015-02-16 21:09 34 ***************
193 2015-02-16 21:10 35 ****************
... ..( 31 skipped). .. ****************
225 2015-02-16 21:42 35 ****************
226 2015-02-16 21:43 36 *****************
227 2015-02-16 21:44 36 *****************
228 2015-02-16 21:45 35 ****************
229 2015-02-16 21:46 34 ***************
230 2015-02-16 21:47 33 **************
231 2015-02-16 21:48 33 **************
232 2015-02-16 21:49 ? -
233 2015-02-16 21:50 33 **************
234 2015-02-16 21:51 32 *************
235 2015-02-16 21:52 32 *************
236 2015-02-16 21:53 33 **************
... ..( 8 skipped). .. **************
245 2015-02-16 22:02 33 **************
246 2015-02-16 22:03 34 ***************
... ..( 3 skipped). .. ***************
250 2015-02-16 22:07 34 ***************
251 2015-02-16 22:08 35 ****************
252 2015-02-16 22:09 35 ****************
SCT Error Recovery Control command not supported
Device Statistics (GP Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x000a 2 9 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x8000 4 14489 Vendor specific
# smartctl -x /dev/sdd
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.18.6-1-ARCH] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: SAMSUNG SpinPoint F3
Device Model: SAMSUNG HD103SJ
Serial Number: S246J9KB924045
LU WWN Device Id: 5 0024e9 2062a5827
Firmware Version: 1AJ10001
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 6
SATA Version is: SATA 2.6, 3.0 Gb/s
Local Time is: Mon Feb 16 22:10:00 2015 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Disabled
APM feature is: Disabled
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 9360) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 156) minutes.
SCT capabilities: (0x003f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 100 100 051 - 1
2 Throughput_Performance -OS--K 055 050 000 - 8667
3 Spin_Up_Time PO---K 069 067 025 - 9424
4 Start_Stop_Count -O--CK 100 100 000 - 589
5 Reallocated_Sector_Ct PO--CK 252 252 010 - 0
7 Seek_Error_Rate -OSR-K 252 252 051 - 0
8 Seek_Time_Performance --S--K 252 252 015 - 0
9 Power_On_Hours -O--CK 100 100 000 - 15097
10 Spin_Retry_Count -O--CK 252 252 051 - 0
11 Calibration_Retry_Count -O--CK 252 252 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 491
191 G-Sense_Error_Rate -O---K 100 100 000 - 3
192 Power-Off_Retract_Count -O---K 252 252 000 - 0
194 Temperature_Celsius -O---- 064 055 000 - 34 (Min/Max 11/45)
195 Hardware_ECC_Recovered -O-RCK 100 100 000 - 0
196 Reallocated_Event_Count -O--CK 252 252 000 - 0
197 Current_Pending_Sector -O--CK 252 252 000 - 0
198 Offline_Uncorrectable ----CK 252 252 000 - 0
199 UDMA_CRC_Error_Count -OS-CK 200 200 000 - 0
200 Multi_Zone_Error_Rate -O-R-K 100 100 000 - 30
223 Load_Retry_Count -O--CK 252 252 000 - 0
225 Load_Cycle_Count -O--CK 100 100 000 - 591
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 2 Comprehensive SMART error log
0x03 GPL R/O 2 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 2 Extended self-test log
0x08 GPL R/O 2 Power Conditions log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (2 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (2 sectors)
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 14928 -
# 2 Extended offline Completed without error 00% 13234 -
# 3 Extended offline Completed without error 00% 5808 -
# 4 Extended offline Interrupted (host reset) 70% 5136 -
SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has
ever been run
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Completed [00% left] (0-65535)
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 2
SCT Version (vendor specific): 256 (0x0100)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 34 Celsius
Power Cycle Min/Max Temperature: 19/34 Celsius
Lifetime Min/Max Temperature: 14/64 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 5 minutes
Temperature Logging Interval: 5 minutes
Min/Max recommended Temperature: -5/80 Celsius
Min/Max Temperature Limit: -10/85 Celsius
Temperature History Size (Index): 128 (7)
Index Estimated Time Temperature Celsius
8 2015-02-16 11:35 35 ****************
9 2015-02-16 11:40 35 ****************
10 2015-02-16 11:45 34 ***************
... ..( 22 skipped). .. ***************
33 2015-02-16 13:40 34 ***************
34 2015-02-16 13:45 35 ****************
35 2015-02-16 13:50 34 ***************
... ..( 22 skipped). .. ***************
58 2015-02-16 15:45 34 ***************
59 2015-02-16 15:50 33 **************
60 2015-02-16 15:55 33 **************
61 2015-02-16 16:00 34 ***************
62 2015-02-16 16:05 33 **************
63 2015-02-16 16:10 34 ***************
64 2015-02-16 16:15 33 **************
... ..( 6 skipped). .. **************
71 2015-02-16 16:50 33 **************
72 2015-02-16 16:55 34 ***************
73 2015-02-16 17:00 33 **************
74 2015-02-16 17:05 33 **************
75 2015-02-16 17:10 33 **************
76 2015-02-16 17:15 34 ***************
77 2015-02-16 17:20 33 **************
... ..( 7 skipped). .. **************
85 2015-02-16 18:00 33 **************
86 2015-02-16 18:05 32 *************
87 2015-02-16 18:10 33 **************
88 2015-02-16 18:15 33 **************
89 2015-02-16 18:20 32 *************
90 2015-02-16 18:25 19 -
91 2015-02-16 18:30 22 ***
92 2015-02-16 18:35 24 *****
93 2015-02-16 18:40 25 ******
94 2015-02-16 18:45 26 *******
95 2015-02-16 18:50 27 ********
96 2015-02-16 18:55 28 *********
97 2015-02-16 19:00 28 *********
98 2015-02-16 19:05 29 **********
99 2015-02-16 19:10 29 **********
100 2015-02-16 19:15 30 ***********
... ..( 3 skipped). .. ***********
104 2015-02-16 19:35 30 ***********
105 2015-02-16 19:40 31 ************
... ..( 5 skipped). .. ************
111 2015-02-16 20:10 31 ************
112 2015-02-16 20:15 32 *************
113 2015-02-16 20:20 30 ***********
114 2015-02-16 20:25 30 ***********
115 2015-02-16 20:30 31 ************
116 2015-02-16 20:35 31 ************
117 2015-02-16 20:40 32 *************
118 2015-02-16 20:45 32 *************
119 2015-02-16 20:50 32 *************
120 2015-02-16 20:55 33 **************
... ..( 7 skipped). .. **************
0 2015-02-16 21:35 33 **************
1 2015-02-16 21:40 34 ***************
... ..( 5 skipped). .. ***************
7 2015-02-16 22:10 34 ***************
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
Device Statistics (GP Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 4 0 Command failed due to ICRC error
0x0002 4 0 R_ERR response for data FIS
0x0003 4 0 R_ERR response for device-to-host data FIS
0x0004 4 0 R_ERR response for host-to-device data FIS
0x0005 4 0 R_ERR response for non-data FIS
0x0006 4 0 R_ERR response for device-to-host non-data FIS
0x0007 4 0 R_ERR response for host-to-device non-data FIS
0x0008 4 0 Device-to-host non-data FIS retries
0x0009 4 16 Transition from drive PhyRdy to drive PhyNRdy
0x000a 4 14 Device-to-host register FISes sent due to a COMRESET
0x000b 4 0 CRC errors within host-to-device FIS
0x000d 4 0 Non-CRC errors within host-to-device FIS
0x000f 4 0 R_ERR response for host-to-device data FIS, CRC
0x0010 4 0 R_ERR response for host-to-device data FIS, non-CRC
0x0012 4 0 R_ERR response for host-to-device non-data FIS, CRC
0x0013 4 0 R_ERR response for host-to-device non-data FIS, non-CRC
0x8e00 4 0 Vendor specific
0x8e01 4 0 Vendor specific
0x8e02 4 0 Vendor specific
0x8e03 4 0 Vendor specific
0x8e04 4 0 Vendor specific
0x8e05 4 0 Vendor specific
0x8e06 4 0 Vendor specific
0x8e07 4 0 Vendor specific
0x8e08 4 0 Vendor specific
0x8e09 4 0 Vendor specific
0x8e0a 4 0 Vendor specific
0x8e0b 4 0 Vendor specific
0x8e0c 4 0 Vendor specific
0x8e0d 4 0 Vendor specific
0x8e0e 4 0 Vendor specific
0x8e0f 4 0 Vendor specific
0x8e10 4 0 Vendor specific
0x8e11 4 0 Vendor specific
No /dev/sde anymore...
> Did you read about and fix any timeout mismatch issue?
I did yes but not sure to understand how to set up 120 second timeout
in sysfs...
The following commands is still valid without 1 of my drives?
mdadm --stop /dev/md126
mdadm --assemble --force --verbose --run /dev/md126 /dev/sd[bcd]3
> Also, your device names have changed. You *must* keep track of which
> one is which "RaidDevice". Specifically, what was "sde" now appears to
> be "sdd". Did you reboot? You know that device names are not
> guaranteed to be consistent from one boot to the next, I hope. Show an
> excerpt from "ls -l /dev/disk/by-id/" with your next report so we know
> which drive serial number has which name.
Now you said it, I'm felling stupid... Yes, I reboot every day and I
don't have trace of previous "mapping" (id -> name).
# ls -l /dev/disk/by-id
total 0
lrwxrwxrwx 1 root root 9 Feb 16 21:16
ata-SAMSUNG_HD103SJ_S246J9KB924045 -> ../../sdd
lrwxrwxrwx 1 root root 10 Feb 16 21:16
ata-SAMSUNG_HD103SJ_S246J9KB924045-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Feb 16 21:16
ata-SAMSUNG_HD103SJ_S246J9KB924045-part2 -> ../../sdd2
lrwxrwxrwx 1 root root 10 Feb 16 21:16
ata-SAMSUNG_HD103SJ_S246J9KB924045-part3 -> ../../sdd3
lrwxrwxrwx 1 root root 10 Feb 16 21:16
ata-SAMSUNG_HD103SJ_S246J9KB924045-part4 -> ../../sdd4
lrwxrwxrwx 1 root root 9 Feb 16 21:16 ata-ST3500320AS_9QM8H48M -> ../../sdb
lrwxrwxrwx 1 root root 10 Feb 16 21:16 ata-ST3500320AS_9QM8H48M-part1
-> ../../sdb1
lrwxrwxrwx 1 root root 10 Feb 16 21:16 ata-ST3500320AS_9QM8H48M-part2
-> ../../sdb2
lrwxrwxrwx 1 root root 10 Feb 16 21:16 ata-ST3500320AS_9QM8H48M-part3
-> ../../sdb3
lrwxrwxrwx 1 root root 9 Feb 16 21:16
ata-WDC_WD5001AALS-00E3A0_WD-WCATR3121296 -> ../../sdc
lrwxrwxrwx 1 root root 10 Feb 16 21:16
ata-WDC_WD5001AALS-00E3A0_WD-WCATR3121296-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Feb 16 21:16
ata-WDC_WD5001AALS-00E3A0_WD-WCATR3121296-part2 -> ../../sdc2
lrwxrwxrwx 1 root root 10 Feb 16 21:16
ata-WDC_WD5001AALS-00E3A0_WD-WCATR3121296-part3 -> ../../sdc3
lrwxrwxrwx 1 root root 11 Feb 16 21:17
md-uuid-6408f077:eefbe269:c173be8e:b17eb4df -> ../../md127
lrwxrwxrwx 1 root root 9 Feb 16 21:16 wwn-0x5000c5001099d533 -> ../../sdb
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x5000c5001099d533-part1 ->
../../sdb1
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x5000c5001099d533-part2 ->
../../sdb2
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x5000c5001099d533-part3 ->
../../sdb3
lrwxrwxrwx 1 root root 9 Feb 16 21:16 wwn-0x50014ee25a457f4e -> ../../sdc
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x50014ee25a457f4e-part1 ->
../../sdc1
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x50014ee25a457f4e-part2 ->
../../sdc2
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x50014ee25a457f4e-part3 ->
../../sdc3
lrwxrwxrwx 1 root root 9 Feb 16 21:16 wwn-0x50024e92062a5827 -> ../../sdd
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x50024e92062a5827-part1 ->
../../sdd1
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x50024e92062a5827-part2 ->
../../sdd2
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x50024e92062a5827-part3 ->
../../sdd3
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x50024e92062a5827-part4 ->
../../sdd4
Thanks,
Dush
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Jes Sorensen @ 2015-02-16 22:49 UTC (permalink / raw)
To: NeilBrown; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <wrfjpp99pp9d.fsf@redhat.com>
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> NeilBrown <neilb@suse.de> writes:
>> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
>> wrote:
>>
>>> Dear All,
>>> Any updates on this issue.
>>
>> Probably the same as:
>>
>> http://marc.info/?l=linux-raid&m=142283560704091&w=2
>
> Hi Neil,
>
> I ran some tests on this one against the latest Linus' tree as of today
> (1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
> your pending 3.20 patches.
>
> I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
> run fio on top of a device while it is resyncing and I fail one of the
> devices.
Since Manibalan mentioned this issue wasn't present in earlier kernels,
I started trying to track down what change caused it.
So far I have been able to reproduce the hang as far back as 3.10.
Cheers,
Jes
^ permalink raw reply
* Re: What are mdadm maintainers to do? (was: desktop disk's error recovery timeouts)
From: NeilBrown @ 2015-02-16 23:49 UTC (permalink / raw)
To: Chris; +Cc: linux-raid
In-Reply-To: <loom.20150216T183419-387@post.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 1067 bytes --]
On Mon, 16 Feb 2015 17:48:50 +0000 (UTC) Chris <email.bug@arcor.de> wrote:
>
> Thank you for the additional information, it calls for action.
>
>
> OK, calling for a solution to stop desktop drives from causing data loss and
> affecting the mdadm reputation:
>
>
> I gather that mdadm could ship with one additional udev rule that calls a
> script to check/set scterc, or falls back to increasing the system timout.
>
> Phil, you mentioned having posted such a script, could you prepare it for
> addition to the mdadm package?
>
>
> Would maintainers be ok with adding such a udev rule and script to the package?
"maintainers" ? Plural? That would be nice.
Unfortunately there is just the one singular me....
There are certainly other contributors who
- answer questions on the list
- provide bug reports
- provide bits of code
and I am very thankful to them. But I haven't found a likely co-maintainer
yet :-(
I'm certainly happy to consider and concrete proposal. The more concrete,
the better.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Chris @ 2015-02-17 7:52 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150217104906.62d36c62@notabene.brown>
NeilBrown <neilb <at> suse.de> writes:
> "maintainers" ? Plural? That would be nice.
> Unfortunately there is just the one singular me....
Yes, as Weedy said, I also refered to distro package maintainers.
If we can come up here with an udev rule and a script to call, then upstream
(you) could include this, and distro maintainers could make smartctl a
suggested or recommended package of the mdadm package.
I certainly have not understood the whole topic yet,
what I just got is, that the script should do something like
the following, and I found some implementation below.
Evererybody please answer with improved versions if you can.
if smartctl tool is available
if scterc is disabled
/usr/sbin/smartctl -l scterc,70,70 ${DEVNAME}
else
if screrc is not available
echo 180 >/sys/block/${DEVNAME}/device/timeout
Found an older implementation that "seems to work fine":
http://article.gmane.org/gmane.linux.raid/44566
>
> contents of udev rule:
> ACTION=="add", SUBSYSTEM=="block", KERNEL=="[sh]d[a-z]",
RUN+="/usr/local/bin/settimeout"
>
>
> contents of /usr/local/bin/settimeout:
> #!/bin/bash
>
> [ "${ACTION}" == "add" ] && {
> /usr/sbin/smartctl -l scterc,70,70 ${DEVNAME} || echo 180 >
/sys/${DEVPATH}/device/timeout
> }
>
> I guess, what is missing, is to connect the HDDs
> with a specific "mdadm" event, instead of running
> for each HDD.
> I'm not sure if this is already possible, since
> some "udev" rules for "md" are already existing.
Let's get this disaster prevention into mdadm, even if just as important
reference experience for solving a more general kernel timeout mismatch
problem "symptom of a more generic issue".
http://article.gmane.org/gmane.linux.raid/44557
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Mikael Abrahamsson @ 2015-02-17 8:48 UTC (permalink / raw)
To: Chris; +Cc: linux-raid
In-Reply-To: <loom.20150217T080345-764@post.gmane.org>
On Tue, 17 Feb 2015, Chris wrote:
> Evererybody please answer with improved versions if you can.
>
> if smartctl tool is available
> if scterc is disabled
> /usr/sbin/smartctl -l scterc,70,70 ${DEVNAME}
> else
> if screrc is not available
> echo 180 >/sys/block/${DEVNAME}/device/timeout
>
> Found an older implementation that "seems to work fine":
Hi,
Generally I like this idea, and I agree that this would be a good idea,
but if I was running raid0 or linear, I might not want scterc to be
enabled.
Also, what would the harm be to always bump the timeout to 180 seconds?
Yes, drives would take longer to be kicked out in case of errors, but if
we're confident in scterc working, wouldn't we want to turn down the
timeout to 10-15 seconds then?
Personally I turn on scterc if available and turn up the timeout to 180
seconds, always, regardless what drives I'm running. I'd rather wait
longer for a drive to be considered dead, than to have drives being kicked
due to some hiccup in the system (controller or drive reset) that might
rectify itself.
So I would suggest turning on scterc and turning up the timeout to 180
seconds as soon as mdadm is installed. This is the best tradeoff I can
come up with between stability and fast drive-dead-detection time.
Here on the list I see people all the time coming in with multiple drives
kicked due to controller resets and other intermittent flukes, I never see
people coming in complaining that it took 30 seconds to detect a drive
error. I doubt there'd be much complaint for 180 seconds. If someone needs
faster detect times then my opinion is that they are in the category who
can be expected to tune this value to their application. 180 seconds works
best for the "larger crowd" using mdadm.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Chris @ 2015-02-17 10:37 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <alpine.DEB.2.02.1502170940010.4007@uplift.swm.pp.se>
Mikael Abrahamsson <swmike <at> swm.pp.se> writes:
> if I was running raid0 or linear, I might not want scterc to be
> enabled.
Good Point.
> Also, what would the harm be to always bump the timeout to 180 seconds?
I don't know why the driver authors chose that linux default,
but the todo with both your points:
if the appearind device is an md member device (mdadm examine?)
if smartctl tool is available
if scterc is disabled in cotaining ${HDD_DEV} AND added device is not
raid0/linear
/usr/sbin/smartctl -l scterc,70,70 ${HDD_DEV}
echo 180 >/sys/block/${HDD_DEV}/device/timeout
^ permalink raw reply
* Re: re-add POLICY
From: Chris @ 2015-02-17 15:09 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <loom.20150216T124230-883@post.gmane.org>
> NeilBrown <neilb <at> suse.de> writes:
>
> > If it doesn't, then maybe you need "POLICY action=spare".
>
> OK, I will test this when the notebook is back in the house.
I could test it on another system.
Without adding a bitmap, it required configring
POLICY domain=default action=spare
and calling
mdadm --udev-rules
but then, after removing and inserting sdc again, only two out of six md
partitions got synced.
To see if there is something wrong, I then added the sdc1 md0 member
manually, and it synced without failure.
So I can't tell why the other partitions did not sync atomatically.
Some of the unsynced partition types are 83 (md0 member), but others
are FD (md7 member) like the automatically synced ones.
linux 3.2.0
mdadm v3.2.5
md7 : active raid1 sdc6[3] sda8[2]
14327680 blocks super 1.2 [3/2] [UU_]
bitmap: 1/1 pages [4KB], 65536KB chunk
md3 : active raid1 sdc8[4] sda10[3]
307011392 blocks super 1.2 [3/2] [UU_]
bitmap: 3/3 pages [12KB], 65536KB chunk
md6 : active raid1 sda7[2]
8695680 blocks super 1.2 [3/1] [_U_]
md1 : active raid1 sda6[3](W) sdb2[1]
19513216 blocks super 1.2 [4/2] [_UU_]
md2 : active raid1 sda9[3](W) sdb3[0]
97590144 blocks super 1.2 [4/2] [U_U_]
md0 : active raid1 sdc1[4] sda5[2](W) sdb1[1]
340672 blocks super 1.2 [4/3] [UUU_]
A partition that did not sync automatically:
/dev/sdc7:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 7a5847cd:be0e8510:8e170bf5:5d40143f
Name : name:2 (local to host name)
Creation Time : Sun Dec 2 21:40:58 2012
Raid Level : raid1
Raid Devices : 4
Avail Dev Size : 195187135 (93.07 GiB 99.94 GB)
Array Size : 97590144 (93.07 GiB 99.93 GB)
Used Dev Size : 195180288 (93.07 GiB 99.93 GB)
Data Offset : 131072 sectors
Super Offset : 8 sectors
State : clean
Device UUID : b1a97d12:965e3d08:059acefb:6ac5b7e3
Update Time : Wed Dec 3 11:23:26 2014
Checksum : ac0ce511 - correct
Events : 382479
Device Role : Active device 1
Array State : AAA. ('A' == active, '.' == missing)
And a corresponding partition that is part of the running array:
/dev/sda9:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 7a5847cd:be0e8510:8e170bf5:5d40143f
Name : name:2 (local to host name)
Creation Time : Sun Dec 2 21:40:58 2012
Raid Level : raid1
Raid Devices : 4
Avail Dev Size : 195182592 (93.07 GiB 99.93 GB)
Array Size : 97590144 (93.07 GiB 99.93 GB)
Used Dev Size : 195180288 (93.07 GiB 99.93 GB)
Data Offset : 131072 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4c191282:80769896:378abe34:aeb01b8d
Flags : write-mostly
Update Time : Mon Feb 16 17:41:54 2015
Checksum : 8cb4794c - correct
Events : 384989
Device Role : Active device 2
Array State : A.A. ('A' == active, '.' == missing)
BTW looking at this data now, it seems to me the superblocks almost support
the clean re-sync / conflict detection I was trying to explain.
a) The removed device 1 does not claim that a member
in the running array (0 and 2) has failed (AAA.)
b) The Events count of device 1 is lower than in the running array.
c) The running array/superblock does not seem to keep
a reference of the Event count when device 1 failed, for additional
security that it has not ben started separately.
But b) and c) may not even be necessary, as starting device 1 separately
would make device 1 claim that 0 and 2 have failed, right?
Regards,
Chris
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Chris Murphy @ 2015-02-17 19:33 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <loom.20150217T080345-764@post.gmane.org>
It's not just mdadm. It likewise affects Btrfs, ZFS, and LVM.
Also, there's a lack of granularity with linux command timer and SCT
ERC applying only to the entire block device, not partitions. So
there's a problem for mixed use cases. For example, two drives, each
with two partitions. sda1 and sdb1 are raid0, and sda2 and sdb2 are
raid1. What's the proper configuration for SCT ERC and the SCSI
command timer?
*shrug* I don't think the automatic udev configuration idea is fail
safe. It sounds too easy for it to automatically cause a
misconfiguration. And it also doesn't at all solve the problem that
there's next to no error reporting to user space. smartd does, but
it's narrow in scope and entirely defers to the hard drive's
self-assessment. There's all sorts of problems that aren't in the
domain of SMART that get reported in dmesg, but there's no method for
gnome-shell or KDE or any DE or even send an email to a sysadmin, as
an early warning. Instead, all too often it's "WTF XFS just corrupted
itself!" meanwhile the real problem has been happening for a week,
dmesg/journal is full of errors indicating the nature of those
problems, but nothing bothered to inform a human being until the file
system face planted.
Chris Murphy
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Adam Goryachev @ 2015-02-17 22:47 UTC (permalink / raw)
To: Chris Murphy, linux-raid
In-Reply-To: <CAJCQCtQvoB0yPPc=RXPPyJhpaFwH0pQTRCRaR_ycTbzHrD6rLw@mail.gmail.com>
On 18/02/15 06:33, Chris Murphy wrote:
> It's not just mdadm. It likewise affects Btrfs, ZFS, and LVM.
>
> Also, there's a lack of granularity with linux command timer and SCT
> ERC applying only to the entire block device, not partitions. So
> there's a problem for mixed use cases. For example, two drives, each
> with two partitions. sda1 and sdb1 are raid0, and sda2 and sdb2 are
> raid1. What's the proper configuration for SCT ERC and the SCSI
> command timer?
Umm, actually I don't know enough to disagree, but I'll ask some
questions which probably shows both the assumptions I've made, and might
help others understand the issue better.
If we enable SCT ERC on every drive that supports it, and we are using
the drive (only) in a RAID0/linear array then what is the downside? As I
understand it, the drive will no longer try for > 120sec to recover the
data stored in the "bad" sector, and instead return an unreadable error
message in a short amount of time (well below 30 seconds) which means
the driver will be able to return a read error to the application (or FS
or MD) and the system as a whole will carry on. If we didn't enable SCT
ERC, then the entire drive would vanish, (because the timeout wasn't
changed for the driver) and the current read and every future read/write
will all fail, and the system will probably crash (well, depending on
the application, FS layout, etc).
So, IMHO, it seems that by default, every SCT ERC capable drive should
have this enabled by default. As a part of error recovery (ie, crap that
really important data stored on those few unreadable sectors) the user
could manually disable SCT ERC and re-attempt to request the data from
the drive (eg, during dd_rescue or similar).
Secondly, changing the timeout for those drives that don't support SCT
ERC, again, it is fairly similar to above, we get the error from the
drive before the timeout, except we will avoid the only possible
downside above (failing to read a very unlikely but possible to read
sector). Again, we will avoid dropping the entire drive, even if all
operations on this drive will stop for a longer period of time, it is
probably better than stopping permanently.
So, IMHO, every non SCT ERC capable drive should have the timeout
extended to 120s/180s or whatever the appropriate time is that (most)
drives will respond within. Leaving only the most extremely brain dead
drives which we simply ridicule on the list and anywhere and everywhere
possible to ensure nobody will ever buy them (or the manufacturer will
fix the problems).
Of course, quite possible I've totally over simplified this, and don't
understand the other repercussions?
> *shrug* I don't think the automatic udev configuration idea is fail
> safe. It sounds too easy for it to automatically cause a
> misconfiguration. And it also doesn't at all solve the problem that
> there's next to no error reporting to user space. smartd does, but
> it's narrow in scope and entirely defers to the hard drive's
> self-assessment. There's all sorts of problems that aren't in the
> domain of SMART that get reported in dmesg, but there's no method for
> gnome-shell or KDE or any DE or even send an email to a sysadmin, as
> an early warning. Instead, all too often it's "WTF XFS just corrupted
> itself!" meanwhile the real problem has been happening for a week,
> dmesg/journal is full of errors indicating the nature of those
> problems, but nothing bothered to inform a human being until the file
> system face planted.
Just because the solution doesn't solve the entire problem, it does
solve a part of the problem, so IMHO, better to solve this part of the
problem, and then discuss/try to find a solution to the rest of the
problem. Unless you have a suggestion which can solve both parts of the
problem? I suppose that a "good" sysadmin should install some sort of
log monitoring software which will alert them to issues, whether that is
via some desktop application/popup or email or something else. The
problem is that most of these issues come from "home" users who will
never setup anything like "log file monitoring" or raid scrubs, or
anything else, so if we do decide upon a generic solution that will work
for almost everybody, then we will still need to rely on the distro
maintainers to implement the solution.
PS, I suppose this is one of the "hide the gory details that nobody
understands" balancing with "provide the information to the user so they
can do something about it". One more generic consideration would be to
have the kernel identify which messages are purely informational/debug
and which are errors. Normal syslog has support for many different
levels, but AFAIK, all kernel messages end up in the same basket.
eg (plugging in and removing a USB drive generated the following log
entries as seen from "dmesg":
[614977.802828] usb 3-3: new high-speed USB device number 5 using xhci_hcd
[614977.822724] usb 3-3: New USB device found, idVendor=0951, idProduct=1665
[614977.822729] usb 3-3: New USB device strings: Mfr=1, Product=2,
SerialNumber=3
[614977.822732] usb 3-3: Product: DataTraveler 2.0
[614977.822735] usb 3-3: Manufacturer: Kingston
[614977.822737] usb 3-3: SerialNumber: 60A44C413CCBFE40AB4FFB3E
[614977.822899] usb 3-3: ep 0x81 - rounding interval to 128 microframes,
ep desc says 255 microframes
[614977.822905] usb 3-3: ep 0x2 - rounding interval to 128 microframes,
ep desc says 255 microframes
[614977.836547] usb-storage 3-3:1.0: USB Mass Storage device detected
[614977.836734] scsi6 : usb-storage 3-3:1.0
[614977.836819] usbcore: registered new interface driver usb-storage
[614978.854080] scsi 6:0:0:0: Direct-Access Kingston DataTraveler
2.0 1.00 PQ: 0 ANSI: 4
[614978.854493] sd 6:0:0:0: Attached scsi generic sg2 type 0
[614978.854658] sd 6:0:0:0: [sdb] 15131636 512-byte logical blocks:
(7.74 GB/7.21 GiB)
[614978.854884] sd 6:0:0:0: [sdb] Write Protect is off
[614978.854888] sd 6:0:0:0: [sdb] Mode Sense: 45 00 00 00
[614978.855085] sd 6:0:0:0: [sdb] Write cache: disabled, read cache:
enabled, doesn't support DPO or FUA
[614978.860015] sdb: sdb1
[614978.860864] sd 6:0:0:0: [sdb] Attached SCSI removable disk
[614979.061474] FAT-fs (sdb1): Volume was not properly unmounted. Some
data may be corrupt. Please run fsck.
[615347.862058] usb 3-3: reset high-speed USB device number 5 using xhci_hcd
[615347.862111] usb 3-3: Device not responding to set address.
[615348.065856] usb 3-3: Device not responding to set address.
[615348.269944] usb 3-3: device not accepting address 5, error -71
[615348.326429] usb 3-3: USB disconnect, device number 5
[615348.334730] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called
with disabled ep ffff88011b1b2600
[615348.334744] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called
with disabled ep ffff88011b1b2640
Of the above, I would suggest most of that is "info" while the following
lines might be warnings:
[614979.061474] FAT-fs (sdb1): Volume was not properly unmounted. Some
data may be corrupt. Please run fsck.
These might be error or critical:
[615347.862058] usb 3-3: reset high-speed USB device number 5 using xhci_hcd
[615347.862111] usb 3-3: Device not responding to set address.
[615348.065856] usb 3-3: Device not responding to set address.
[615348.269944] usb 3-3: device not accepting address 5, error -71
Of course, this will rely on every driver maintainer to make a decision
on just how important each line that they log may be.
Just my thoughts, hopefully it will be useful.
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Chris @ 2015-02-17 23:33 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAJCQCtQvoB0yPPc=RXPPyJhpaFwH0pQTRCRaR_ycTbzHrD6rLw@mail.gmail.com>
Chris Murphy writes:
>
> It's not just mdadm. It likewise affects Btrfs, ZFS, and LVM.
Do they have own timouts, or rely on the kernel?
Maybe the kernel could read the SCTERT value from the drives (in lieu of
some better retry timout information, and set the controller timout a little
greater than that, or very large if SCTERT is disabled/not available.
> sda1 and sdb1 are raid0, and sda2 and sdb2 are
> raid1. What's the proper configuration for SCT ERC and the SCSI
> command timer?
guessing...
For SCTERT disabled drives:
A compromise may be to stay with the linux default controller timout, it's
30s, and set the drives SCTERT below 30s (maybe 27s), to avoid losing
redundancy and risking data loss *AND* allow more of the available time for ERC.
For longer error correcting attempts (and just as long i/o controller
blocking!) the contoller timout could be set to 180s, and SCTERT to 175s?
BUT: If I chose to use a raid0 alongside a redundant raid I already
explicitly decided to take all data loss the hardware throws at me. So I
don't think it makes much of a difference if ERC times out after <30 secs or
180s, its just more or less errors belonging to me.
For SCTERC enabled drives:
30s and 7s seems ok?
> *shrug* I don't think the automatic udev configuration idea is fail
> safe. It sounds too easy for it to automatically cause a
> misconfiguration.
A matching timeout configuration prevents that unavoidable unrecoverable
read error take down the redundancy for sure, and cause high risk of data
loss during rebuild.
It does fix a misconfiguration, however could possibly set SCTERT just below
the (30s) controler timout, to reduce the impact of SCTERT (e.g make use of
the small chance of error correction succceding a couple of seconds later).
Given the longer SCTERT timout does not lead to subseqent read error timouts
piling up.
> And it also doesn't at all solve the problem that
> there's next to no error reporting to user space.
That is correct, but rather not related to the importance to fix the timout
mismatch and reduce the risk, is it? The settings do solve unecessary loss
of redundancy on read errors that are sure to occur, unnecessary resyncing,
and high risk of data loss during all that.
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Jes Sorensen @ 2015-02-18 0:03 UTC (permalink / raw)
To: NeilBrown; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <wrfjd259o4iy.fsf@redhat.com>
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> NeilBrown <neilb@suse.de> writes:
>>> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
>>> wrote:
>>>
>>>> Dear All,
>>>> Any updates on this issue.
>>>
>>> Probably the same as:
>>>
>>> http://marc.info/?l=linux-raid&m=142283560704091&w=2
>>
>> Hi Neil,
>>
>> I ran some tests on this one against the latest Linus' tree as of today
>> (1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
>> your pending 3.20 patches.
>>
>> I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
>> run fio on top of a device while it is resyncing and I fail one of the
>> devices.
>
> Since Manibalan mentioned this issue wasn't present in earlier kernels,
> I started trying to track down what change caused it.
>
> So far I have been able to reproduce the hang as far back as 3.10.
After a lot of bisecting I finally traced the issue back to this commit:
a7854487cd7128a30a7f4f5259de9f67d5efb95f is the first bad commit
commit a7854487cd7128a30a7f4f5259de9f67d5efb95f
Author: Alexander Lyakas <alex.bolshoy@gmail.com>
Date: Thu Oct 11 13:50:12 2012 +1100
md: When RAID5 is dirty, force reconstruct-write instead of read-modify-write.
Signed-off-by: Alex Lyakas <alex@zadarastorage.com>
Suggested-by: Yair Hershko <yair@zadarastorage.com>
Signed-off-by: NeilBrown <neilb@suse.de>
If I revert that one I cannot reproduce the hang, applying it reproduces
the hang consistently.
Cheers,
Jes
^ permalink raw reply
* [PATCH] Revert "md: When RAID5 is dirty, force reconstruct-write instead of read-modify-write."
From: Jes.Sorensen @ 2015-02-18 0:14 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, Jes Sorensen
From: Jes Sorensen <Jes.Sorensen@redhat.com>
This reverts commit a7854487cd7128a30a7f4f5259de9f67d5efb95f.
The above patch would cause lockups of RAID4/5 arrays if a drive is
going faulty during reconstruction.
Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
drivers/md/raid5.c | 19 +++----------------
1 file changed, 3 insertions(+), 16 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index aa76865..bc3a085 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -3159,25 +3159,12 @@ static void handle_stripe_dirtying(struct r5conf *conf,
int disks)
{
int rmw = 0, rcw = 0, i;
- sector_t recovery_cp = conf->mddev->recovery_cp;
-
- /* RAID6 requires 'rcw' in current implementation.
- * Otherwise, check whether resync is now happening or should start.
- * If yes, then the array is dirty (after unclean shutdown or
- * initial creation), so parity in some stripes might be inconsistent.
- * In this case, we need to always do reconstruct-write, to ensure
- * that in case of drive failure or read-error correction, we
- * generate correct data from the parity.
- */
- if (conf->max_degraded == 2 ||
- (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
- /* Calculate the real rcw later - for now make it
+ if (conf->max_degraded == 2) {
+ /* RAID6 requires 'rcw' in current implementation
+ * Calculate the real rcw later - for now fake it
* look like rcw is cheaper
*/
rcw = 1; rmw = 2;
- pr_debug("force RCW max_degraded=%u, recovery_cp=%llu sh->sector=%llu\n",
- conf->max_degraded, (unsigned long long)recovery_cp,
- (unsigned long long)sh->sector);
} else for (i = disks; i--; ) {
/* would I have to read this buffer for read_modify_write */
struct r5dev *dev = &sh->dev[i];
--
1.8.3.1
^ permalink raw reply related
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: NeilBrown @ 2015-02-18 0:27 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <wrfj61b0gk6l.fsf@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 3998 bytes --]
On Tue, 17 Feb 2015 19:03:30 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
wrote:
> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> > Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> >> NeilBrown <neilb@suse.de> writes:
> >>> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
> >>> wrote:
> >>>
> >>>> Dear All,
> >>>> Any updates on this issue.
> >>>
> >>> Probably the same as:
> >>>
> >>> http://marc.info/?l=linux-raid&m=142283560704091&w=2
> >>
> >> Hi Neil,
> >>
> >> I ran some tests on this one against the latest Linus' tree as of today
> >> (1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
> >> your pending 3.20 patches.
> >>
> >> I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
> >> run fio on top of a device while it is resyncing and I fail one of the
> >> devices.
> >
> > Since Manibalan mentioned this issue wasn't present in earlier kernels,
> > I started trying to track down what change caused it.
> >
> > So far I have been able to reproduce the hang as far back as 3.10.
>
> After a lot of bisecting I finally traced the issue back to this commit:
>
> a7854487cd7128a30a7f4f5259de9f67d5efb95f is the first bad commit
> commit a7854487cd7128a30a7f4f5259de9f67d5efb95f
> Author: Alexander Lyakas <alex.bolshoy@gmail.com>
> Date: Thu Oct 11 13:50:12 2012 +1100
>
> md: When RAID5 is dirty, force reconstruct-write instead of read-modify-write.
>
> Signed-off-by: Alex Lyakas <alex@zadarastorage.com>
> Suggested-by: Yair Hershko <yair@zadarastorage.com>
> Signed-off-by: NeilBrown <neilb@suse.de>
>
> If I revert that one I cannot reproduce the hang, applying it reproduces
> the hang consistently.
Thanks for all the research!
That is consistent with what you already reported.
You noted that it doesn't affect RAID6, and RAID6 doesn't have an RMW cycle.
Also, one of the early emails from Manibalan contained:
handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1
, check:0, reconstruct:0
check 5: state 0x10 read (null) write (null) written (null)
check 4: state 0x11 read (null) write (null) written (null)
check 3: state 0x0 read (null) write (null) written (null)
check 2: state 0x11 read (null) write (null) written (null)
check 1: state 0x11 read (null) write (null) written (null)
check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1
force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328
for sector 273480328, rmw=2 rcw=1
So it is forcing RCW, even though a single block update is usually handled
with RMW.
In this stripe, the parity disk is '5' and disk 3 has failed.
That means to perform an RCW, we need to read the parity block in order
to reconstruct the content of the failed disk. And if we were to do that,
we may as well just do an RMW.
So I think the correct fix would be to only force RCW when the array
is not degraded.
So something like this:
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index aa76865b804b..fa8f8b94bfa8 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -3170,7 +3170,8 @@ static void handle_stripe_dirtying(struct r5conf *conf,
* generate correct data from the parity.
*/
if (conf->max_degraded == 2 ||
- (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
+ (recovery_cp < MaxSector && sh->sector >= recovery_cp &&
+ s->failed == 0)) {
/* Calculate the real rcw later - for now make it
* look like rcw is cheaper
*/
I think reverting the whole patch is not necessary and discards useful
functionality while the array is not degraded.
Can you test this patch please?
Thanks!
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply related
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Jes Sorensen @ 2015-02-18 1:01 UTC (permalink / raw)
To: NeilBrown; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <20150218112741.08495514@notabene.brown>
NeilBrown <neilb@suse.de> writes:
> On Tue, 17 Feb 2015 19:03:30 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
> wrote:
>
>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> > Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> >> NeilBrown <neilb@suse.de> writes:
>> >>> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
>> >>> wrote:
>> >>>
>> >>>> Dear All,
>> >>>> Any updates on this issue.
>> >>>
>> >>> Probably the same as:
>> >>>
>> >>> http://marc.info/?l=linux-raid&m=142283560704091&w=2
>> >>
>> >> Hi Neil,
>> >>
>> >> I ran some tests on this one against the latest Linus' tree as of today
>> >> (1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
>> >> your pending 3.20 patches.
>> >>
>> >> I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
>> >> run fio on top of a device while it is resyncing and I fail one of the
>> >> devices.
>> >
>> > Since Manibalan mentioned this issue wasn't present in earlier kernels,
>> > I started trying to track down what change caused it.
>> >
>> > So far I have been able to reproduce the hang as far back as 3.10.
>>
>> After a lot of bisecting I finally traced the issue back to this commit:
>>
>> a7854487cd7128a30a7f4f5259de9f67d5efb95f is the first bad commit
>> commit a7854487cd7128a30a7f4f5259de9f67d5efb95f
>> Author: Alexander Lyakas <alex.bolshoy@gmail.com>
>> Date: Thu Oct 11 13:50:12 2012 +1100
>>
>> md: When RAID5 is dirty, force reconstruct-write instead of
>> read-modify-write.
>>
>> Signed-off-by: Alex Lyakas <alex@zadarastorage.com>
>> Suggested-by: Yair Hershko <yair@zadarastorage.com>
>> Signed-off-by: NeilBrown <neilb@suse.de>
>>
>> If I revert that one I cannot reproduce the hang, applying it reproduces
>> the hang consistently.
>
> Thanks for all the research!
>
> That is consistent with what you already reported.
> You noted that it doesn't affect RAID6, and RAID6 doesn't have an RMW cycle.
>
> Also, one of the early emails from Manibalan contained:
>
> handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1
> , check:0, reconstruct:0
> check 5: state 0x10 read (null) write (null) written (null)
> check 4: state 0x11 read (null) write (null) written (null)
> check 3: state 0x0 read (null) write (null) written (null)
> check 2: state 0x11 read (null) write (null) written (null)
> check 1: state 0x11 read (null) write (null) written (null)
> check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1
> force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328
> for sector 273480328, rmw=2 rcw=1
>
> So it is forcing RCW, even though a single block update is usually handled
> with RMW.
>
> In this stripe, the parity disk is '5' and disk 3 has failed.
> That means to perform an RCW, we need to read the parity block in order
> to reconstruct the content of the failed disk. And if we were to do that,
> we may as well just do an RMW.
>
> So I think the correct fix would be to only force RCW when the array
> is not degraded.
>
> So something like this:
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index aa76865b804b..fa8f8b94bfa8 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -3170,7 +3170,8 @@ static void handle_stripe_dirtying(struct r5conf *conf,
> * generate correct data from the parity.
> */
> if (conf->max_degraded == 2 ||
> - (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
> + (recovery_cp < MaxSector && sh->sector >= recovery_cp &&
> + s->failed == 0)) {
> /* Calculate the real rcw later - for now make it
> * look like rcw is cheaper
> */
>
>
> I think reverting the whole patch is not necessary and discards useful
> functionality while the array is not degraded.
>
> Can you test this patch please?
Actually I just tried this one - I was on my way home and grabbed food
on the way, and thought there was a better solution than to revert.
I'll give your solution a spin too.
Jes
From 63e7c81c955d99e1eb7ab70956689a12e02eb856 Mon Sep 17 00:00:00 2001
From: Jes Sorensen <Jes.Sorensen@redhat.com>
Date: Tue, 17 Feb 2015 19:48:49 -0500
Subject: [PATCH] [md] raid5.c: Do not force reconstruct writes on a degraded
array
If an array has no writeable spares, do not try to force
reconstruct-write to it.
Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
drivers/md/raid5.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index aa76865..c0036c4 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -3170,7 +3170,8 @@ static void handle_stripe_dirtying(struct r5conf *conf,
* generate correct data from the parity.
*/
if (conf->max_degraded == 2 ||
- (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
+ (conf->mddev->degraded < (conf->max_degraded - 1) &&
+ recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
/* Calculate the real rcw later - for now make it
* look like rcw is cheaper
*/
--
2.1.0
^ permalink raw reply related
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Chris Murphy @ 2015-02-18 1:02 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <54E3C51C.2080106@websitemanagers.com.au>
On Tue, Feb 17, 2015 at 3:47 PM, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
> If we enable SCT ERC on every drive that supports it, and we are using the
> drive (only) in a RAID0/linear array then what is the downside?
Unnecessary data loss.
> As I
> understand it, the drive will no longer try for > 120sec to recover the data
> stored in the "bad" sector, and instead return an unreadable error message
> in a short amount of time (well below 30 seconds) which means the driver
> will be able to return a read error to the application (or FS or MD) and the
> system as a whole will carry on.
Not necessarily, it depends what's in that sector. If it's user data,
this means a sector (or possibly more) of data loss. If it's file
system metadata it means progressive file system corruption.
Configuring the drive to give up too soon is completely inappropriate
for single, raid0 or linear configurations.
Arguably the drive should have already recovered this data. If a
longer recovery can recover, then why isn't the drive writing the data
back to that sector so that next time it isn't so ambiguous that it
requires long recovery? I can't answer that question. In some case
that appears to happen in other cases it's not. But the followup is
that there really ought to be some way for user space to get access to
these kinds of errors rather than them accumulating until disaster
strikes.
The contra argument to that is, it's still cheaper to buy the proper
use case specified drive.
>If we didn't enable SCT ERC, then the
> entire drive would vanish, (because the timeout wasn't changed for the
> driver) and the current read and every future read/write will all fail, and
> the system will probably crash (well, depending on the application, FS
> layout, etc).
Umm no. If SCT ERC remains a high value or disable, while also
increasing the kernel command timer, the drive has a longer chance to
recover. That's the appropriate configuration for single, linear, and
raid0.
>
> So, IMHO, it seems that by default, every SCT ERC capable drive should have
> this enabled by default. As a part of error recovery (ie, crap that really
> important data stored on those few unreadable sectors) the user could
> manually disable SCT ERC and re-attempt to request the data from the drive
> (eg, during dd_rescue or similar).
If you do this for single, linear, or raid0 it will increase the
incident of data loss that would otherwise not occur if deep/long
recovery times were available.
Before changing these settings, there should be some better
understanding of what the manufacturer defined recovery times in the
real world actually are, and whether or not these long recoveries are
helpful. Presumably they'd say they are helpful, but I think we need
facts to contradict their position before second guessing the default
settings. And we have such facts to do exactly that when it comes to
raid1, 5, 6 with such drives which is why the recommendation is to
change SCT ERC if supported.
> Secondly, changing the timeout for those drives that don't support SCT ERC,
> again, it is fairly similar to above, we get the error from the drive before
> the timeout, except we will avoid the only possible downside above (failing
> to read a very unlikely but possible to read sector). Again, we will avoid
> dropping the entire drive, even if all operations on this drive will stop
> for a longer period of time, it is probably better than stopping
> permanently.
Not by default. You can't assume any drive hang is due to bad sectors
that merely need a longer recovery time. It could be some other error
condition, in which case doing a 120 or 180 second *by default* delay
means no error messages at all for upwards of 3 minutes.
And in any case the proper place to change the default kernel command
timer value is in the kernel, not with a udev rule.
I don't know if a udev rule can say "If the drive exclusively uses md,
lvm, btrfs, zfs raid1, 4+ or nested of those, and if the drive does
not support configurable SCT ERC, then change the kernel command timer
for those devices to ~120 seconds" then that might be a plausible
solution to use consumer drives the manufacturer rather explicitly
proscribes from use in raid...
But the contra argument to that is, why should anyone do this work for
(sorry) basically cheap users who don't want to buy the proper drive
for the specific use case? There are limited resources for this work.
And in fact the problem has a work around, if not a solution.
What we still don't have is something that reports any such problems
to user space.
--
Chris Murphy
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Jes Sorensen @ 2015-02-18 1:07 UTC (permalink / raw)
To: NeilBrown; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <wrfjh9ukf2y6.fsf@redhat.com>
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> NeilBrown <neilb@suse.de> writes:
>> On Tue, 17 Feb 2015 19:03:30 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
>> wrote:
>>
>>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>>> > Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>>> >> NeilBrown <neilb@suse.de> writes:
>>> >>> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
>>> >>> wrote:
>>> >>>
>>> >>>> Dear All,
>>> >>>> Any updates on this issue.
>>> >>>
>>> >>> Probably the same as:
>>> >>>
>>> >>> http://marc.info/?l=linux-raid&m=142283560704091&w=2
>>> >>
>>> >> Hi Neil,
>>> >>
>>> >> I ran some tests on this one against the latest Linus' tree as of today
>>> >> (1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
>>> >> your pending 3.20 patches.
>>> >>
>>> >> I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
>>> >> run fio on top of a device while it is resyncing and I fail one of the
>>> >> devices.
>>> >
>>> > Since Manibalan mentioned this issue wasn't present in earlier kernels,
>>> > I started trying to track down what change caused it.
>>> >
>>> > So far I have been able to reproduce the hang as far back as 3.10.
>>>
>>> After a lot of bisecting I finally traced the issue back to this commit:
>>>
>>> a7854487cd7128a30a7f4f5259de9f67d5efb95f is the first bad commit
>>> commit a7854487cd7128a30a7f4f5259de9f67d5efb95f
>>> Author: Alexander Lyakas <alex.bolshoy@gmail.com>
>>> Date: Thu Oct 11 13:50:12 2012 +1100
>>>
>>> md: When RAID5 is dirty, force reconstruct-write instead of
>>> read-modify-write.
>>>
>>> Signed-off-by: Alex Lyakas <alex@zadarastorage.com>
>>> Suggested-by: Yair Hershko <yair@zadarastorage.com>
>>> Signed-off-by: NeilBrown <neilb@suse.de>
>>>
>>> If I revert that one I cannot reproduce the hang, applying it reproduces
>>> the hang consistently.
>>
>> Thanks for all the research!
>>
>> That is consistent with what you already reported.
>> You noted that it doesn't affect RAID6, and RAID6 doesn't have an RMW cycle.
>>
>> Also, one of the early emails from Manibalan contained:
>>
>> handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1
>> , check:0, reconstruct:0
>> check 5: state 0x10 read (null) write (null) written (null)
>> check 4: state 0x11 read (null) write (null) written (null)
>> check 3: state 0x0 read (null) write (null) written (null)
>> check 2: state 0x11 read (null) write (null) written (null)
>> check 1: state 0x11 read (null) write (null) written (null)
>> check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
>> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1
>> force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328
>> for sector 273480328, rmw=2 rcw=1
>>
>> So it is forcing RCW, even though a single block update is usually handled
>> with RMW.
>>
>> In this stripe, the parity disk is '5' and disk 3 has failed.
>> That means to perform an RCW, we need to read the parity block in order
>> to reconstruct the content of the failed disk. And if we were to do that,
>> we may as well just do an RMW.
>>
>> So I think the correct fix would be to only force RCW when the array
>> is not degraded.
>>
>> So something like this:
>>
>> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
>> index aa76865b804b..fa8f8b94bfa8 100644
>> --- a/drivers/md/raid5.c
>> +++ b/drivers/md/raid5.c
>> @@ -3170,7 +3170,8 @@ static void handle_stripe_dirtying(struct r5conf *conf,
>> * generate correct data from the parity.
>> */
>> if (conf->max_degraded == 2 ||
>> - (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
>> + (recovery_cp < MaxSector && sh->sector >= recovery_cp &&
>> + s->failed == 0)) {
>> /* Calculate the real rcw later - for now make it
>> * look like rcw is cheaper
>> */
>>
>>
>> I think reverting the whole patch is not necessary and discards useful
>> functionality while the array is not degraded.
>>
>> Can you test this patch please?
>
> Actually I just tried this one - I was on my way home and grabbed food
> on the way, and thought there was a better solution than to revert.
>
> I'll give your solution a spin too.
I tried your patch, as expected that also resolves the problem. Not sure
which solution is better, so I'll let you pick.
Note whichever patch you choose it is applicable for stable-3.6+
Cheers,
Jes
^ permalink raw reply
* Re: [PATCH] Revert "md: When RAID5 is dirty, force reconstruct-write instead of read-modify-write."
From: Jes Sorensen @ 2015-02-18 1:08 UTC (permalink / raw)
To: neilb; +Cc: linux-raid
In-Reply-To: <1424218444-4022-1-git-send-email-Jes.Sorensen@redhat.com>
Jes.Sorensen@redhat.com writes:
> From: Jes Sorensen <Jes.Sorensen@redhat.com>
>
> This reverts commit a7854487cd7128a30a7f4f5259de9f67d5efb95f.
>
> The above patch would cause lockups of RAID4/5 arrays if a drive is
> going faulty during reconstruction.
>
> Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
> ---
> drivers/md/raid5.c | 19 +++----------------
> 1 file changed, 3 insertions(+), 16 deletions(-)
Please ignore this one - per previous email, there is a better way<tm>
Cheers,
Jes
>
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index aa76865..bc3a085 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -3159,25 +3159,12 @@ static void handle_stripe_dirtying(struct r5conf *conf,
> int disks)
> {
> int rmw = 0, rcw = 0, i;
> - sector_t recovery_cp = conf->mddev->recovery_cp;
> -
> - /* RAID6 requires 'rcw' in current implementation.
> - * Otherwise, check whether resync is now happening or should start.
> - * If yes, then the array is dirty (after unclean shutdown or
> - * initial creation), so parity in some stripes might be inconsistent.
> - * In this case, we need to always do reconstruct-write, to ensure
> - * that in case of drive failure or read-error correction, we
> - * generate correct data from the parity.
> - */
> - if (conf->max_degraded == 2 ||
> - (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
> - /* Calculate the real rcw later - for now make it
> + if (conf->max_degraded == 2) {
> + /* RAID6 requires 'rcw' in current implementation
> + * Calculate the real rcw later - for now fake it
> * look like rcw is cheaper
> */
> rcw = 1; rmw = 2;
> - pr_debug("force RCW max_degraded=%u, recovery_cp=%llu sh->sector=%llu\n",
> - conf->max_degraded, (unsigned long long)recovery_cp,
> - (unsigned long long)sh->sector);
> } else for (i = disks; i--; ) {
> /* would I have to read this buffer for read_modify_write */
> struct r5dev *dev = &sh->dev[i];
^ permalink raw reply
* Re: RAID1 might_sleep() warning on 3.19-rc7
From: NeilBrown @ 2015-02-18 1:09 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: Tony Battersby, linux-raid, lkml, axboe, Linus Torvalds
In-Reply-To: <20150213144803.GA11574@worktop.ger.corp.intel.com>
[-- Attachment #1: Type: text/plain, Size: 907 bytes --]
On Fri, 13 Feb 2015 15:48:03 +0100 Peter Zijlstra <peterz@infradead.org>
wrote:
> On Fri, Feb 13, 2015 at 11:27:46AM +0100, Peter Zijlstra wrote:
>
> > > I've moved blk_flush_plug to the beginning of the function.
> >
> > > I wondered if it really make sense to call blk_flush_plug with nr_iowait
> > > elevated and delayacct_blkio active. blk_flush_plug() could call schedule()
> > > for non-"io" reasons and maybe that could upset stuff???
> >
> > Yeah, good question that. Lemme ponder that a bit.
>
> Yes, I thikn your version makes most sense as, you say, even regular
> schedule() call nested in my version would go towards blk delayacct --
> and I doubt that was the intent; even though the current kernel works
> that way.
>
> I'll move the now rudimentary io_schedule() into sched.h as an inline.
Thanks.
Are you OK with this going to -stable for 3.19?
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: NeilBrown @ 2015-02-18 1:16 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <wrfjsie4do37.fsf@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 5172 bytes --]
On Tue, 17 Feb 2015 20:07:24 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
wrote:
> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> > NeilBrown <neilb@suse.de> writes:
> >> On Tue, 17 Feb 2015 19:03:30 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
> >> wrote:
> >>
> >>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> >>> > Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> >>> >> NeilBrown <neilb@suse.de> writes:
> >>> >>> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
> >>> >>> wrote:
> >>> >>>
> >>> >>>> Dear All,
> >>> >>>> Any updates on this issue.
> >>> >>>
> >>> >>> Probably the same as:
> >>> >>>
> >>> >>> http://marc.info/?l=linux-raid&m=142283560704091&w=2
> >>> >>
> >>> >> Hi Neil,
> >>> >>
> >>> >> I ran some tests on this one against the latest Linus' tree as of today
> >>> >> (1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
> >>> >> your pending 3.20 patches.
> >>> >>
> >>> >> I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
> >>> >> run fio on top of a device while it is resyncing and I fail one of the
> >>> >> devices.
> >>> >
> >>> > Since Manibalan mentioned this issue wasn't present in earlier kernels,
> >>> > I started trying to track down what change caused it.
> >>> >
> >>> > So far I have been able to reproduce the hang as far back as 3.10.
> >>>
> >>> After a lot of bisecting I finally traced the issue back to this commit:
> >>>
> >>> a7854487cd7128a30a7f4f5259de9f67d5efb95f is the first bad commit
> >>> commit a7854487cd7128a30a7f4f5259de9f67d5efb95f
> >>> Author: Alexander Lyakas <alex.bolshoy@gmail.com>
> >>> Date: Thu Oct 11 13:50:12 2012 +1100
> >>>
> >>> md: When RAID5 is dirty, force reconstruct-write instead of
> >>> read-modify-write.
> >>>
> >>> Signed-off-by: Alex Lyakas <alex@zadarastorage.com>
> >>> Suggested-by: Yair Hershko <yair@zadarastorage.com>
> >>> Signed-off-by: NeilBrown <neilb@suse.de>
> >>>
> >>> If I revert that one I cannot reproduce the hang, applying it reproduces
> >>> the hang consistently.
> >>
> >> Thanks for all the research!
> >>
> >> That is consistent with what you already reported.
> >> You noted that it doesn't affect RAID6, and RAID6 doesn't have an RMW cycle.
> >>
> >> Also, one of the early emails from Manibalan contained:
> >>
> >> handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1
> >> , check:0, reconstruct:0
> >> check 5: state 0x10 read (null) write (null) written (null)
> >> check 4: state 0x11 read (null) write (null) written (null)
> >> check 3: state 0x0 read (null) write (null) written (null)
> >> check 2: state 0x11 read (null) write (null) written (null)
> >> check 1: state 0x11 read (null) write (null) written (null)
> >> check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
> >> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1
> >> force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328
> >> for sector 273480328, rmw=2 rcw=1
> >>
> >> So it is forcing RCW, even though a single block update is usually handled
> >> with RMW.
> >>
> >> In this stripe, the parity disk is '5' and disk 3 has failed.
> >> That means to perform an RCW, we need to read the parity block in order
> >> to reconstruct the content of the failed disk. And if we were to do that,
> >> we may as well just do an RMW.
> >>
> >> So I think the correct fix would be to only force RCW when the array
> >> is not degraded.
> >>
> >> So something like this:
> >>
> >> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> >> index aa76865b804b..fa8f8b94bfa8 100644
> >> --- a/drivers/md/raid5.c
> >> +++ b/drivers/md/raid5.c
> >> @@ -3170,7 +3170,8 @@ static void handle_stripe_dirtying(struct r5conf *conf,
> >> * generate correct data from the parity.
> >> */
> >> if (conf->max_degraded == 2 ||
> >> - (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
> >> + (recovery_cp < MaxSector && sh->sector >= recovery_cp &&
> >> + s->failed == 0)) {
> >> /* Calculate the real rcw later - for now make it
> >> * look like rcw is cheaper
> >> */
> >>
> >>
> >> I think reverting the whole patch is not necessary and discards useful
> >> functionality while the array is not degraded.
> >>
> >> Can you test this patch please?
> >
> > Actually I just tried this one - I was on my way home and grabbed food
> > on the way, and thought there was a better solution than to revert.
> >
> > I'll give your solution a spin too.
>
> I tried your patch, as expected that also resolves the problem. Not sure
> which solution is better, so I'll let you pick.
Thanks!
>
> Note whichever patch you choose it is applicable for stable-3.6+
3.6??
$ git describe --contains a7854487cd7128a30a7f4f5259
v3.7-rc1~10^2~7
so I assume 3.7.
Doesn't apply to 3.6, so I'll assume a typo.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* [GIT PULL REQUEST] some more bugfixes for md.
From: NeilBrown @ 2015-02-18 1:31 UTC (permalink / raw)
To: Linus Torvalds; +Cc: lkml, linux RAID, Nate Dailey, Jes Sorensen, Manibalan P
[-- Attachment #1: Type: text/plain, Size: 1174 bytes --]
hi Linus,
Please pull these three fixes, one tagged for -stable.
Thanks,
NeilBrown
The following changes since commit 53a6ab4d3f6d6dc87ec8f14998b4b5536ee2968c:
md/raid10: fix conversion from RAID0 to RAID10 (2015-02-12 14:09:57 +1100)
are available in the git repository at:
git://neil.brown.name/md/ tags/md/3.20-fixes
for you to fetch changes up to 26ac107378c4742978216be1005b7291b799c7b2:
md/raid5: Fix livelock when array is both resyncing and degraded. (2015-02-18 11:35:14 +1100)
----------------------------------------------------------------
3 bug md fixes for 3.20
yet-another-livelock in raid5, and a problem with write errors
to 4K-block devices.
----------------------------------------------------------------
Nate Dailey (1):
md/raid1: round up to bdev_logical_block_size in narrow_write_error
NeilBrown (2):
md/raid10: round up to bdev_logical_block_size in narrow_write_error.
md/raid5: Fix livelock when array is both resyncing and degraded.
drivers/md/raid1.c | 3 ++-
drivers/md/raid10.c | 3 ++-
drivers/md/raid5.c | 3 ++-
3 files changed, 6 insertions(+), 3 deletions(-)
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [PATCH 0/3] md raid: enhancements to support the device mapper dm-raid target
From: NeilBrown @ 2015-02-18 2:03 UTC (permalink / raw)
To: heinzm; +Cc: linux-raid, dm-devel
In-Reply-To: <1423853282-6218-1-git-send-email-heinzm@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 2701 bytes --]
On Fri, 13 Feb 2015 19:47:59 +0100 heinzm@redhat.com wrote:
> From: Heinz Mauelshagen <heinzm@redhat.com>
>
> I'm enhancing the device mapper raid target (dm-raid) to take
> advantage of so far unused md raid kernel funtionality:
> takeover, reshape, resize, addition and removal of devices to/from raid sets.
>
> This series of patches remove constraints doing so.
>
>
> Patch #1:
> add 2 API functions to allow dm-raid to access the raid takeover
> and resize functionality (namely md_takeover() and md_resize());
> reshape APIs are not needed in lieu of the existing personalilty ones
>
> Patch #2:
> because device mapper core manages a request queue per mapped device
> utilizing the md make_request API to pass on bios via the dm-raid target,
> no md instance underneath it needs to manage a request queue of its own.
> Thus dm-raid can't use the md raid0 personality as is, because the latter
> accesses the request queue unconditionally in 3 places via mddev->queue
> which this patch addresses.
>
> Patch #3:
> when dm-raid processes a down takeover to raid0, it needs to destroy
> any existing bitmap, because raid0 does not require one. The patch
> exports the bitmap_destroy() API to allow dm-raid to remove bitmaps.
>
>
> Heinz Mauelshagen (3):
> md core: add 2 API functions for takeover and resize to support dm-raid
> md raid0: access mddev->queue (request queue member) conditionally
> because it is not set when accessed from dm-raid
> md bitmap: export bitmap_destroy() to support dm-raid down takover to raid0
>
> drivers/md/bitmap.c | 1 +
> drivers/md/md.c | 39 ++++++++++++++++++++++++++++++---------
> drivers/md/md.h | 3 +++
> drivers/md/raid0.c | 48 +++++++++++++++++++++++++++---------------------
> 4 files changed, 61 insertions(+), 30 deletions(-)
>
Hi Heinz,
I don't object to these patches if you will find the exported functionality
useful, but I am a little surprised by them.
I would expect that dm-raid wouldn't ask md to 'takeover' from one level to
another, but instead would
- suspend the dm device
- dismantle the array using the old level
- assemble the array using the new level
- resume the dm device
The reason md needs 'takeover' is because it doesn't have the same
device/target separation that dm does.
I was particularly surprised that you wanted to use md/raid0.c It is no
better than dm/dm-stripe.c and managing two different stripe engines under
LVM doesn't see like a good idea.
Is there some reason that I have missed which makes it easier to use
'takeover' rather than suspend/resume?
Thanks,
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Jes Sorensen @ 2015-02-18 5:05 UTC (permalink / raw)
To: NeilBrown; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <20150218121656.0584e09d@notabene.brown>
NeilBrown <neilb@suse.de> writes:
> On Tue, 17 Feb 2015 20:07:24 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
> wrote:
>
>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> > NeilBrown <neilb@suse.de> writes:
>> >> On Tue, 17 Feb 2015 19:03:30 -0500 Jes Sorensen <Jes.Sorensen@redhat.com>
>> >> wrote:
>> >>
>> >>> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> >>> > Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> >>> >> NeilBrown <neilb@suse.de> writes:
>> >>> >>> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P
>> >>> >>> <pmanibalan@amiindia.co.in>
>> >>> >>> wrote:
>> >>> >>>
>> >>> >>>> Dear All,
>> >>> >>>> Any updates on this issue.
>> >>> >>>
>> >>> >>> Probably the same as:
>> >>> >>>
>> >>> >>> http://marc.info/?l=linux-raid&m=142283560704091&w=2
>> >>> >>
>> >>> >> Hi Neil,
>> >>> >>
>> >>> >> I ran some tests on this one against the latest Linus' tree as of today
>> >>> >> (1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
>> >>> >> your pending 3.20 patches.
>> >>> >>
>> >>> >> I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
>> >>> >> run fio on top of a device while it is resyncing and I fail one of the
>> >>> >> devices.
>> >>> >
>> >>> > Since Manibalan mentioned this issue wasn't present in earlier kernels,
>> >>> > I started trying to track down what change caused it.
>> >>> >
>> >>> > So far I have been able to reproduce the hang as far back as 3.10.
>> >>>
>> >>> After a lot of bisecting I finally traced the issue back to this commit:
>> >>>
>> >>> a7854487cd7128a30a7f4f5259de9f67d5efb95f is the first bad commit
>> >>> commit a7854487cd7128a30a7f4f5259de9f67d5efb95f
>> >>> Author: Alexander Lyakas <alex.bolshoy@gmail.com>
>> >>> Date: Thu Oct 11 13:50:12 2012 +1100
>> >>>
>> >>> md: When RAID5 is dirty, force reconstruct-write instead of
>> >>> read-modify-write.
>> >>>
>> >>> Signed-off-by: Alex Lyakas <alex@zadarastorage.com>
>> >>> Suggested-by: Yair Hershko <yair@zadarastorage.com>
>> >>> Signed-off-by: NeilBrown <neilb@suse.de>
>> >>>
>> >>> If I revert that one I cannot reproduce the hang, applying it reproduces
>> >>> the hang consistently.
>> >>
>> >> Thanks for all the research!
>> >>
>> >> That is consistent with what you already reported.
>> >> You noted that it doesn't affect RAID6, and RAID6 doesn't have an RMW cycle.
>> >>
>> >> Also, one of the early emails from Manibalan contained:
>> >>
>> >> handling stripe 273480328, state=0x2041 cnt=1, pd_idx=5, qd_idx=-1
>> >> , check:0, reconstruct:0
>> >> check 5: state 0x10 read (null) write (null) written (null)
>> >> check 4: state 0x11 read (null) write (null) written (null)
>> >> check 3: state 0x0 read (null) write (null) written (null)
>> >> check 2: state 0x11 read (null) write (null) written (null)
>> >> check 1: state 0x11 read (null) write (null) written (null)
>> >> check 0: state 0x18 read (null) write ffff8808029b6b00 written (null)
>> >> locked=0 uptodate=3 to_read=0 to_write=1 failed=1 failed_num=3,-1
>> >> force RCW max_degraded=1, recovery_cp=7036944 sh->sector=273480328
>> >> for sector 273480328, rmw=2 rcw=1
>> >>
>> >> So it is forcing RCW, even though a single block update is usually handled
>> >> with RMW.
>> >>
>> >> In this stripe, the parity disk is '5' and disk 3 has failed.
>> >> That means to perform an RCW, we need to read the parity block in order
>> >> to reconstruct the content of the failed disk. And if we were to do that,
>> >> we may as well just do an RMW.
>> >>
>> >> So I think the correct fix would be to only force RCW when the array
>> >> is not degraded.
>> >>
>> >> So something like this:
>> >>
>> >> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
>> >> index aa76865b804b..fa8f8b94bfa8 100644
>> >> --- a/drivers/md/raid5.c
>> >> +++ b/drivers/md/raid5.c
>> >> @@ -3170,7 +3170,8 @@ static void handle_stripe_dirtying(struct r5conf *conf,
>> >> * generate correct data from the parity.
>> >> */
>> >> if (conf->max_degraded == 2 ||
>> >> - (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
>> >> + (recovery_cp < MaxSector && sh->sector >= recovery_cp &&
>> >> + s->failed == 0)) {
>> >> /* Calculate the real rcw later - for now make it
>> >> * look like rcw is cheaper
>> >> */
>> >>
>> >>
>> >> I think reverting the whole patch is not necessary and discards useful
>> >> functionality while the array is not degraded.
>> >>
>> >> Can you test this patch please?
>> >
>> > Actually I just tried this one - I was on my way home and grabbed food
>> > on the way, and thought there was a better solution than to revert.
>> >
>> > I'll give your solution a spin too.
>>
>> I tried your patch, as expected that also resolves the problem. Not sure
>> which solution is better, so I'll let you pick.
>
> Thanks!
>
>>
>> Note whichever patch you choose it is applicable for stable-3.6+
>
> 3.6??
>
> $ git describe --contains a7854487cd7128a30a7f4f5259
> v3.7-rc1~10^2~7
>
> so I assume 3.7.
> Doesn't apply to 3.6, so I'll assume a typo.
Too many kernels in one day, you win - 3.7 it is :)
Jes
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Chris @ 2015-02-18 11:04 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAJCQCtTv-HMX77GWqA+_1rfkWodsTETp_Y3w5En-N4nGCymWsA@mail.gmail.com>
>
Hello all,
the discussion about SCTERC boils down to letting the drive attempt ERC a
little more or less. For any given disk experience seems to tell the slight
difference is, that if ERC is allowed longer you may see the first
unrecoverable erros (UREs) just a little (maybe only a month) later.
UREs are inevitable. Thus, if I run a filesystem on just a single drive it
will get corrupted at some point, nothing to do about it.
Wait, except..., use a redundant raid! And here it makes a lot of a
difference that the drive's ERC actually terminates before the controller
timeout, to not loose all your redundacy again and be in hight risk of UREs
showing up during the re-sync.
So for a proper comparison we need to look at the difference it makes in the
usage scenarios (error delay vs. loosing redundant error resilence + URE
triggering), not at the single recoverable/unrecoverable error incidence. It
looks to me, that it makes a lot of a differnce to redundant raids and no
qualitative difference to single disk filesystems.
And we need to keep in mind that single disk filesystems do also depend on
the disk to stop grinding away with ERC attempts before the controller
timout. Otherwise disk reset may make the system clear buffers and loose
open files? Without prolonging the linux default controller timout, SCTERC
can prevent that where supported.
> in any case the proper place to change the default kernel command
> timer value is in the kernel, not with a udev rule.
Right. And as you write increasing the controller timout has clear downsides.
Noteing as well, as long as the proposed script (a temporary safety measure)
maximizes the controller timeout to remedy for disks that don's support
SCTERC, this would even fix the timout mismatch for single disk filesystems.
(Letting the controller wait until the disk finally succeeds or fails its
recovery attempts.)
So the proposed script actually provides a case that brings benefit for
raid0 setups as well (as long as the linux default is not adaptive to the
disk parameters), but increasing the controller timout in all cases would
introduce long and unreported i/o blocking into all redundant setups.
> I don't know if a udev rule can say "If the drive exclusively uses md,
> lvm, btrfs, zfs raid1, 4+ or nested of those, and if the drive does
> not support configurable SCT ERC, then change the kernel command timer
> for those devices to ~120 seconds" then that might be a plausible
> solution to use consumer drives the manufacturer rather explicitly
> proscribes from use in raid...
The script called by the udev rule could do that, but can be kept as simple
as proposed, and can set SCTERC regardles, because setting SCTERC below the
controller timout makes a qualitative difference in running the redundant
arrays and a marginal difference in running non-redundant filesystems. (And
nevertheless, set long controller timout for devices that don's support SCTERC.)
After all, this looks like a quite simple change is appropriate:
In udev-md-raid-assembly.rules, below LABEL="md_inc" (only handling all md
suppported devices) add one rule:
# fix timouts for redundant raids, if possible
TEST="/usr/sbin/smartctl", ENV{MD_LEVEL}=="raid[1-9]*",
RUN+="/usr/bin/mdadm-erc-timout-fix"
And in a new /usr/bin/mdadm-erc-timout-fix file implement:
if smartctl -l scterc ${HDD_DEV} returns "Disabled"
/usr/sbin/smartctl -l scterc,70,70 ${HDD_DEV}
else
if smartctl -l scterc ${HDD_DEV} does not return "seconds"
echo 180 >/sys/block/${HDD_DEV}/device/timeout
Regards,
Chris
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox