* Re: mdadm failed to remove internal bitmap
From: gary @ 2015-02-16 6:42 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <20150216143419.553cf93e@notabene.brown>
Hi Neil,
Please check the followings.
> What kernel are you running?
linux48:~ # uname -r
3.12.32-33-default
And 3.12.28-4-default kernel is ok.
> Please use "strace" on mdadm in a case where it fails, and post the result.
linux48:~ # strace mdadm --grow --bitmap=none /dev/md127
execve("/sbin/mdadm", ["mdadm", "--grow", "--bitmap=none",
"/dev/md127"], [/* 58 vars */]) = 0
brk(0) = 0xfe8000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fcab07ad000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or
directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=93919, ...}) = 0
mmap(NULL, 93919, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fcab0796000
close(3) = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\34\2\0\0\0\0\0"...,
832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1978611, ...}) = 0
mmap(NULL, 3832352, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x7fcab01e6000
mprotect(0x7fcab0384000, 2097152, PROT_NONE) = 0
mmap(0x7fcab0584000, 24576, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19e000) = 0x7fcab0584000
mmap(0x7fcab058a000, 14880, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fcab058a000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fcab0795000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fcab0794000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x7fcab0793000
arch_prctl(ARCH_SET_FS, 0x7fcab0794700) = 0
mprotect(0x7fcab0584000, 16384, PROT_READ) = 0
mprotect(0x677000, 4096, PROT_READ) = 0
mprotect(0x7fcab07ae000, 4096, PROT_READ) = 0
munmap(0x7fcab0796000, 93919) = 0
getpid() = 6000
brk(0) = 0xfe8000
brk(0x1009000) = 0x1009000
open("/dev/md127", O_RDWR) = 3
fstat(3, {st_mode=S_IFBLK|0660, st_rdev=makedev(9, 127), ...}) = 0
ioctl(3, RAID_VERSION, 0x7fff2e10d160) = 0
open("/etc/mdadm.conf", O_RDONLY) = -1 ENOENT (No such file or
directory)
open("/etc/mdadm/mdadm.conf", O_RDONLY) = -1 ENOENT (No such file or
directory)
open("/etc/mdadm.conf.d", O_RDONLY) = -1 ENOENT (No such file or
directory)
uname({sys="Linux", node="linux48", ...}) = 0
geteuid() = 0
fstat(3, {st_mode=S_IFBLK|0660, st_rdev=makedev(9, 127), ...}) = 0
ioctl(3, RAID_VERSION, 0x7fff2e10c060) = 0
ioctl(3, GET_BITMAP_FILE, 0x7fff2e10c1f0) = 0
ioctl(3, GET_ARRAY_INFO, 0x7fff2e10c1a0) = 0
ioctl(3, SET_ARRAY_INFO, 0x7fff2e10c1a0) = -1 EINVAL (Invalid argument)
write(2, "mdadm: failed to remove internal"..., 41mdadm: failed to
remove internal bitmap.
) = 41
exit_group(1) = ?
+++ exited with 1 +++
Thanks,
gary
^ permalink raw reply
* Re: mdadm failed to remove internal bitmap
From: NeilBrown @ 2015-02-16 6:48 UTC (permalink / raw)
To: gary; +Cc: linux-raid
In-Reply-To: <54E1914C.9030000@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]
On Mon, 16 Feb 2015 14:42:20 +0800 gary <gary.mdjiang@gmail.com> wrote:
> Hi Neil,
>
> Please check the followings.
> > What kernel are you running?
> linux48:~ # uname -r
> 3.12.32-33-default
Does this have any patches on top of 3.12.32 that touch md.c ?
>
> And 3.12.28-4-default kernel is ok.
There are no differences between 3.12.28 and 3.12.32 that could affect this.
> ioctl(3, RAID_VERSION, 0x7fff2e10c060) = 0
> ioctl(3, GET_BITMAP_FILE, 0x7fff2e10c1f0) = 0
> ioctl(3, GET_ARRAY_INFO, 0x7fff2e10c1a0) = 0
> ioctl(3, SET_ARRAY_INFO, 0x7fff2e10c1a0) = -1 EINVAL (Invalid argument)
> write(2, "mdadm: failed to remove internal"..., 41mdadm: failed to
> remove internal bitmap.
EINVAL from SET_ARRAY_INFO almost certainly comes from update_array_info().
It can happen if:
- more than 1 thing needs to be updated - seems unlikely
- pers->quiesce is NULL - not possible for raid1.
- mddev->bitmap->storage.file is not NULL. Seems unlikely.
I suggest you look at the code you are actually running, and possible add
some printks to tell you where it is failing.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* RE: RAID 1 metadata - keep separate from mirror disks ?
From: Suresh Babu Kandukuru @ 2015-02-16 11:37 UTC (permalink / raw)
To: Phil Turmel, linux-raid; +Cc: Ankur Bose
In-Reply-To: <54DB6BF0.3040309@turmel.org>
Yeh . Thank Phil . it is quite useful . Now we see there are two options . 1) without metadata 2) externally managed metadata . But we would like to keep the metadata external to mirror leg ( like on host local drive ) for the reasons mentioned below , not the externally managed metadata , just by keeping metadata on mirror legs
. Do we have any option with md driver ?
/Suresh
Principal developer | Hyderabad Team Lead
Phone: +914067246370 | Mobile: +919701451727
Oracle Maxrep development
ORACLE India Hyderabad
Oracle is committed to developing practices and products that help protect the environment
-----Original Message-----
From: Phil Turmel [mailto:philip@turmel.org]
Sent: Wednesday, February 11, 2015 8:19 PM
To: Suresh Babu Kandukuru; linux-raid@vger.kernel.org
Subject: Re: RAID 1 metadata - keep separate from mirror disks ?
Good morning Suresh,
On 02/11/2015 07:14 AM, Suresh Babu Kandukuru wrote:
> Hi There,
>
> On the RAID 1 metadata: is there any way to keep the metadata
> separate from the mirror disks? Could you guide us on this ?,
> please. In general, we need to keep all metadata off the device
> itself, leaving all the device available for user data. This is
> particularly important in the migration case, where we want to take an
> existing LUN and add a second leg to it to create the mirror device
> without changing any of the data or metadata on the LUN.
If you look at "man 4 md" you'll see some options. If a legacy array type meets your needs, you can operate without metadata at all. Use "mdadm --build" to assemble your raid at each boot.
Or, if your storage server can insert a leg ahead of you current LUN, you can then create the array with an explicit data offset matching the size of the inserted leg. Create it degraded with the existing LUN, then add (a) LUN(s) to start mirroring. This process will leave you the option to resize with more legs later.
Or you can add a leg to the end and create your array with version 1.0 metadata, which is placed at the end of the device.
Finally, you could write your own metadata container service for use with mdmon. (That's a bit beyond my ability, sorry.)
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: re-add POLICY
From: Chris @ 2015-02-16 12:23 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150216142845.0d50207c@notabene.brown>
NeilBrown <neilb <at> suse.de> writes:
> Does your array have a write-intent bitmap configured?
> If it does, then "POLICY action=re-add" really should work.
Thank you for your insight. You are correct, the array has no write-intent
bitmap.
> If it doesn't, then maybe you need "POLICY action=spare".
OK, I will test this when the notebook is back in the house.
Actually, the man page had kind of kept me from trying this, because it
mentions the condition "if the device is bare", and I didn't want arbitrary
bare disk, partition, or free space to be automatically added, but just to
trigger an automatic try with raid members that got pulled and are save to
re-sync. (e.g. after the occasional bad block error that gets remapped by
the hardrives firmware)
[man page: spare works] "as above and additionally: if the device is
bare it can become a spare if there is any array that it is a candidate for
based on domains and metadata."
Also, I wouldn't want a temporarily removed raid member to be added as spare
to some other array. Only have them added (re-synced even if no bitmap
re-add is possible) to the array they belong according to their superblock.
> This isn't the default, because depending on exactly how/why the device
> failed, it may not be safe to treat it as a spare.
OK, I can imagine detecting the corner cases may require some inteligent
error logging.
What I am looking for is a safe re-sync configuration option between
bitmap-based re-add, and treating a device as arbitrary spare drive.
Practically, this could be something like an additional action=re-sync
option in between re-add/spare, or having the "re-add" action also do
(non-bitmap) full re-syncs, if the device is in a clean state.
May recording the fail event count in the remaining superblocks help, as
described in
http://permalink.gmane.org/gmane.linux.raid/48077
help to detect the clean state?
Kind Regards,
Chris
^ permalink raw reply
* Re: RAID 1 metadata - keep separate from mirror disks ?
From: Phil Turmel @ 2015-02-16 13:02 UTC (permalink / raw)
To: Suresh Babu Kandukuru, linux-raid; +Cc: Ankur Bose
In-Reply-To: <17135927-75bb-499a-8f43-6748c872feb0@default>
Hi Suresh,
On 02/16/2015 06:37 AM, Suresh Babu Kandukuru wrote:
> Yeh . Thank Phil . it is quite useful . Now we see there are two
> options . 1) without metadata 2) externally managed metadata . But we
> would like to keep the metadata external to mirror leg ( like on host
> local drive ) for the reasons mentioned below , not the externally
> managed metadata , just by keeping metadata on mirror legs . Do we
> have any option with md driver ?
I'm not sure I understand your follow-up question. You've restated my
answer, and re-iterated your preference to not have metadata on the
member devices.
So you need to write a service to handle metadata the way you want. Or
write scripts that will do --build operations at appropriate times
without metadata.
I'm not entirely clear why the on-member metadata is unacceptable, but
as such, either remaining option needs some code of your own.
Phil
^ permalink raw reply
* Re: re-add POLICY
From: Phil Turmel @ 2015-02-16 13:17 UTC (permalink / raw)
To: Chris, linux-raid
In-Reply-To: <loom.20150216T124230-883@post.gmane.org>
Hi Chris,
On 02/16/2015 07:23 AM, Chris wrote:
> .... with raid members that got pulled and are save to
> re-sync. (e.g. after the occasional bad block error that gets remapped by
> the hardrives firmware)
This should not be part of your concern here, as MD will handle
occassional UREs by reconstructing them and rewriting them on the fly,
-- without failing the device. If devices are failing after read
errors, you have a different problem. (Hint: look at recent threads
for "timeout mismatch".)
Phil
^ permalink raw reply
* desktop disk's error recovery timouts (was: re-add POLICY)
From: Chris @ 2015-02-16 16:15 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <54E1EDEA.1030503@turmel.org>
Phil Turmel <philip <at> turmel.org> writes:
> On 02/16/2015 07:23 AM, Chris wrote:
> > .... with raid members that got pulled and are save to
> > re-sync. (e.g. after the occasional bad block error that gets remapped by
> > the hardrives firmware)
>
> This should not be part of your concern here, as MD will handle
> occassional UREs by reconstructing them and rewriting them on the fly,
Phil, thank you for dropping in with this hint. It very likly applies to
the disks in the docking station. I searched the mailing list, most hits
said to search for the keywords, though. ;-)
To understand the issue, I think
https://en.wikipedia.org/wiki/Error_recovery_control
was good.
It would be good if this configuration information could be available there
or at https://raid.wiki.kernel.org
Cheers,
Chris
----
I compiled some snippets from your messages, that could serve as a basis to
correction/completion by someone knowledgeable:
The default linux controller timeout is 30 seconds. Drives
that spend longer than the timeout in recovery will be reset. If they
don't respond to the reset (because they're busy in recovery) when the
raid tries to write the correct data back to them, they will be kicked
out of the array.
You *must* set ERC shorter than the
timeout, or set the driver timeout longer than the drive's worst-case
recovery time. The defaults for desktop drives are *not* suitable for
linux software raid.
I strongly encourage you to run "smartctl -l scterc /dev/sdX" for each
of your drives. For any drive that warns that it doesn't support SCT
ERC, set the controller device timeout to 180 like so:
echo 180 >/sys/block/sdX/device/timeout
If the report says read or write ERC is disabled, run "smartctl -l
scterc,70,70 /dev/sdX" to set it to 7.0 seconds.
You then set up a boot-time script to do these adjustments at every restart,
and make sure you performing regular scrub runs to ...?
You might not want that kind of long device timeout, but then you shouldn't
use desktop drives in md RAID.
Anyone using desktop drives which don't support SCT ERC in md RAID is
liable to see long timeouts on the simplest bad sector, and they
probably prefer to keep the drive in the array AND have the sector
rewritten after reconstruction than have the drive failed out of the array.
^ permalink raw reply
* Re: desktop disk's error recovery timouts
From: Phil Turmel @ 2015-02-16 17:19 UTC (permalink / raw)
To: Chris, linux-raid
In-Reply-To: <loom.20150216T165111-551@post.gmane.org>
On 02/16/2015 11:15 AM, Chris wrote:
> Phil, thank you for dropping in with this hint. It very likly applies to
> the disks in the docking station. I searched the mailing list, most hits
> said to search for the keywords, though. ;-)
I don't always have time to explain. :-(
> To understand the issue, I think
> https://en.wikipedia.org/wiki/Error_recovery_control
> was good.
Good starting points in the archives:
http://marc.info/?l=linux-raid&m=135811522817345&w=1
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=135863964624202&w=2
http://marc.info/?l=linux-raid&m=139050322510249&w=2
There's useful info in each entire thread, though.
Phil
^ permalink raw reply
* What are mdadm maintainers to do? (was: desktop disk's error recovery timeouts)
From: Chris @ 2015-02-16 17:48 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <54E226B5.1080500@turmel.org>
Thank you for the additional information, it calls for action.
OK, calling for a solution to stop desktop drives from causing data loss and
affecting the mdadm reputation:
I gather that mdadm could ship with one additional udev rule that calls a
script to check/set scterc, or falls back to increasing the system timout.
Phil, you mentioned having posted such a script, could you prepare it for
addition to the mdadm package?
Would maintainers be ok with adding such a udev rule and script to the package?
Kind Regards,
Chris
^ permalink raw reply
* Re: All drive in Raid 5 are in 'spare' mode
From: Dush @ 2015-02-16 19:38 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <54DC04C1.4010107@turmel.org>
Hi Phil,
Thanks for your answer!
Unfortunately, I think I just loosed a disk (sde)... I don't see it
anymore in /dev , I have in dmesg:
d
[ 12.280021] ata7: softreset failed (1st FIS failed)
[ 22.280019] ata7: softreset failed (1st FIS failed)
[ 57.280015] ata7: softreset failed (1st FIS failed)
[ 57.280222] ata7: limiting SATA link speed to 1.5 Gbps
[ 62.453345] ata7: softreset failed (device not ready)
[ 62.453558] ata7: reset failed, giving up
And my reports look like this now:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md126 : inactive sdb3[3](S) sdd3[1](S) sdc3[0](S)
1447416000 blocks
md127 : active (auto-read-only) raid5 sdd2[1] sdb2[3] sdc2[0]
16530624 blocks level 5, 64k chunk, algorithm 2 [4/3] [UU_U]
unused devices: <none>
# mdadm --examine /dev/sd[b-e]3
/dev/sdb3:
Magic : a92b4efc
Version : 0.90.00
UUID : 3327f442:a00b59b2:1397f3c2:236c0edf
Creation Time : Tue Jan 27 13:03:52 2009
Raid Level : raid5
Used Dev Size : 482472000 (460.12 GiB 494.05 GB)
Array Size : 1447416000 (1380.36 GiB 1482.15 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 126
Update Time : Wed Jan 21 20:55:48 2015
State : active
Active Devices : 3
Working Devices : 4
Failed Devices : 1
Spare Devices : 1
Checksum : 6e656c69 - correct
Events : 49656
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 19 3 active sync /dev/sdb3
0 0 8 35 0 active sync /dev/sdc3
1 1 8 67 1 active sync
2 2 0 0 2 faulty removed
3 3 8 19 3 active sync /dev/sdb3
4 4 8 51 4 spare /dev/sdd3
/dev/sdc3:
Magic : a92b4efc
Version : 0.90.00
UUID : 3327f442:a00b59b2:1397f3c2:236c0edf
Creation Time : Tue Jan 27 13:03:52 2009
Raid Level : raid5
Used Dev Size : 482472000 (460.12 GiB 494.05 GB)
Array Size : 1447416000 (1380.36 GiB 1482.15 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 126
Update Time : Wed Jan 21 23:34:52 2015
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 2
Spare Devices : 1
Checksum : 6e6653d5 - correct
Events : 49666
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 35 0 active sync /dev/sdc3
0 0 8 35 0 active sync /dev/sdc3
1 1 8 67 1 active sync
2 2 0 0 2 faulty removed
3 3 0 0 3 faulty removed
4 4 8 51 4 spare /dev/sdd3
/dev/sdd3:
Magic : a92b4efc
Version : 0.90.00
UUID : 3327f442:a00b59b2:1397f3c2:236c0edf
Creation Time : Tue Jan 27 13:03:52 2009
Raid Level : raid5
Used Dev Size : 482472000 (460.12 GiB 494.05 GB)
Array Size : 1447416000 (1380.36 GiB 1482.15 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 126
Update Time : Wed Jan 21 23:34:52 2015
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 2
Spare Devices : 1
Checksum : 6e6653f7 - correct
Events : 49666
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 67 1 active sync
0 0 8 35 0 active sync /dev/sdc3
1 1 8 67 1 active sync
2 2 0 0 2 faulty removed
3 3 0 0 3 faulty removed
4 4 8 51 4 spare /dev/sdd3
You was right, I already tried to start the raid and it succeed to do
it with 3 drives: b, c and e. Then I added the d because I thought it
was de-synchronized.
Now I think my drive e was out of this raid for a while and I started
to had trouble because d started to had some issues.
Is it possible to force raid to start with b, c and d (forcing d to be
'normal')? Time for me to copy everything to another drive...
Thanks,
Dush
On 12 February 2015 at 01:41, Phil Turmel <philip@turmel.org> wrote:
> Hi Dush,
>
> On 02/11/2015 02:56 PM, Dush wrote:
>> Hi,
>>
>> I have a RAID 5 composed by 4x 500Go hdd but for some days, it's 'inactive'.
>>
>> I'm not raid expert and I prefer asking before doing an unrecoverable mistake...
>>
>> Is it possible to fix this raid (md126)?
>> Is it possible to recover data on it?
>
> Probably. Very good report, btw.
>
>> Do I have a disk to change or it's "just" a desynchronization between disks?
>
> One disk is now truly a spare (/dev/sdd3), which suggests you already
> tried to '--add' it and didn't get anywhere.
>
> Step one: collect some forensics for later. syslog or dmesg containing
> your failure events. Can be trimmed to just device and md stuff.
> "smartctl -x /dev/sdX" for each drive involved in the arrays.
>
> Then, we'll try the simple stuff.
>
> Make sure the array is stopped with:
>
> mdadm --stop /dev/md126
>
> Then, force assemble it without sdd:
>
> mdadm --assemble --force --verbose --run /dev/md126 /dev/sd[bce]3
>
> If that works, mount it and catch a backup of critical files.
>
> Then add your /dev/sdd3 back to the array and let it rebuild:
>
> mdadm --add /dev/md126 /dev/sdd3
>
> It may not make it through the rebuild if you have the common timeout
> mismatch problem.[1] Show the dmesg and smartctl data (pasted inline is
> preferred) and we'll see.
>
> Phil
>
> Recent typical case:
> [1] http://marc.info/?l=linux-raid&m=142353387024935&w=1
>
^ permalink raw reply
* Re: What are mdadm maintainers to do?
From: Phil Turmel @ 2015-02-16 19:44 UTC (permalink / raw)
To: Chris, linux-raid
In-Reply-To: <loom.20150216T183419-387@post.gmane.org>
On 02/16/2015 12:48 PM, Chris wrote:
>
> Thank you for the additional information, it calls for action.
>
>
> OK, calling for a solution to stop desktop drives from causing data loss and
> affecting the mdadm reputation:
>
>
> I gather that mdadm could ship with one additional udev rule that calls a
> script to check/set scterc, or falls back to increasing the system timout.
>
> Phil, you mentioned having posted such a script, could you prepare it for
> addition to the mdadm package?
No, I've posted snippets for users to customize in their own rc.local or
distro equivalent. I vaguely recall posting a generic script for some
common cases, but I've personally converted to raid-rated drives
everywhere in the past couple years.
Somebody else will have to tackle this.
> Would maintainers be ok with adding such a udev rule and script to the package?
Not my call, but keep in mind that this will add a dependency on
smartmontools or whatever means is used to access/write to scterc.
Phil
^ permalink raw reply
* Re: All drive in Raid 5 are in 'spare' mode
From: Phil Turmel @ 2015-02-16 20:03 UTC (permalink / raw)
To: Dush; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <CAL7hTOfhWiCmKVVCtpmHuB-OJkZjQsuaKZMN8oFKAArh=NuSHQ@mail.gmail.com>
Hi Dush,
{Convention on kernel.org is to trim replies and either bottom post or
interleave. Please don't top-post.}
On 02/16/2015 02:38 PM, Dush wrote:
> Hi Phil,
>
> Thanks for your answer!
>
> Unfortunately, I think I just loosed a disk (sde)... I don't see it
> anymore in /dev , I have in dmesg:
> d
> [ 12.280021] ata7: softreset failed (1st FIS failed)
> [ 22.280019] ata7: softreset failed (1st FIS failed)
> [ 57.280015] ata7: softreset failed (1st FIS failed)
> [ 57.280222] ata7: limiting SATA link speed to 1.5 Gbps
> [ 62.453345] ata7: softreset failed (device not ready)
> [ 62.453558] ata7: reset failed, giving up
Where are the forensics I asked for as "Step one"? Did you read about
and fix any timeout mismatch issue?
[trim /]
> You was right, I already tried to start the raid and it succeed to do
> it with 3 drives: b, c and e. Then I added the d because I thought it
> was de-synchronized.
> Now I think my drive e was out of this raid for a while and I started
> to had trouble because d started to had some issues.
>
> Is it possible to force raid to start with b, c and d (forcing d to be
> 'normal')? Time for me to copy everything to another drive...
No. sdd3 was converted to a spare.
Also, your device names have changed. You *must* keep track of which
one is which "RaidDevice". Specifically, what was "sde" now appears to
be "sdd". Did you reboot? You know that device names are not
guaranteed to be consistent from one boot to the next, I hope. Show an
excerpt from "ls -l /dev/disk/by-id/" with your next report so we know
which drive serial number has which name.
Phil
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Jes Sorensen @ 2015-02-16 20:36 UTC (permalink / raw)
To: NeilBrown; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <20150203093040.569aa5e1@notabene.brown>
NeilBrown <neilb@suse.de> writes:
> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
> wrote:
>
>> Dear All,
>> Any updates on this issue.
>
> Probably the same as:
>
> http://marc.info/?l=linux-raid&m=142283560704091&w=2
Hi Neil,
I ran some tests on this one against the latest Linus' tree as of today
(1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
your pending 3.20 patches.
I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
run fio on top of a device while it is resyncing and I fail one of the
devices.
I can reproduce the issue for raid4 and raid5, but I don't see it if I
I use a raid6.
The following sequence consistently reproduces the problem for me:
mdadm -C /dev/md111 -f -e 1.2 -l5 -n4 /dev/sd[ghij]3
fio --name=md111 --filename=/dev/md111 --thread --numjobs=10 --direct=1 --group_reporting --unlink=0 --loops=1 --offset=0 --randrepeat=1 --norandommap --scramble_buffers=1 --stonewall --rw=randwrite --bs=8704 --iodepth=4000 --runtime=3000 --blockalign=512
mdadm /dev/md111 -f /dev/sdg3
Cheers,
Jes
^ permalink raw reply
* Re: All drive in Raid 5 are in 'spare' mode
From: Dush @ 2015-02-16 22:24 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <54E24D10.7030901@turmel.org>
> {Convention on kernel.org is to trim replies and either bottom post or
> interleave. Please don't top-post.}
Sorry about that...
> Where are the forensics I asked for as "Step one"?
# dmesg
[ 12.273354] ata7: softreset failed (1st FIS failed)
[ 22.273352] ata7: softreset failed (1st FIS failed)
[ 57.273349] ata7: softreset failed (1st FIS failed)
[ 57.273556] ata7: limiting SATA link speed to 1.5 Gbps
[ 62.446679] ata7: softreset failed (device not ready)
[ 62.446892] ata7: reset failed, giving up
[ 62.933371] ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 62.939016] ata8.00: ATA-8: SAMSUNG HD103SJ, 1AJ10001, max UDMA/133
[ 62.939019] ata8.00: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA
[ 62.944696] ata8.00: configured for UDMA/133
[ 0.380968] random: systemd-tmpfile urandom read with 1 bits of
entropy available
[ 0.411310] ata1: PATA max UDMA/100 cmd 0xdc00 ctl 0xd880 bmdma 0xd400 irq 17
[ 0.411312] ata2: PATA max UDMA/100 cmd 0xd800 ctl 0xd480 bmdma 0xd408 irq 17
[ 63.473837] systemd[1]: Cannot add dependency job for unit
gssproxy.service, ignoring: Unit gssproxy.service failed to load: No
such file or directory.
[ 63.632259] systemd-journald[157]: Received request to flush
runtime journal from PID 1
[ 63.884029] md: bind<sdc2>
[ 63.895136] md: bind<sdc3>
[ 63.910832] md: bind<sdb2>
[ 63.926903] md: bind<sdd3>
[ 63.937151] md: bind<sdd2>
[ 63.940172] md: bind<sdb3>
[ 94.538293] md: raid6 personality registered for level 6
[ 94.538298] md: raid5 personality registered for level 5
[ 94.538300] md: raid4 personality registered for level 4
[ 94.538954] md/raid:md127: device sdd2 operational as raid disk 1
[ 94.538958] md/raid:md127: device sdb2 operational as raid disk 3
[ 94.538960] md/raid:md127: device sdc2 operational as raid disk 0
[ 94.539457] md/raid:md127: allocated 0kB
[ 94.539493] md/raid:md127: raid level 5 active with 3 out of 4
devices, algorithm 2
[ 94.539568] md127: detected capacity change from 0 to 16927358976
[ 94.544725] md127: unknown partition table
# smartctl -x /dev/sdb
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.18.6-1-ARCH] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.11
Device Model: ST3500320AS
Serial Number: 9QM8H48M
LU WWN Device Id: 5 000c50 01099d533
Firmware Version: SD15
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 2.6, 3.0 Gb/s
Local Time is: Mon Feb 16 22:09:04 2015 GMT
==> WARNING: There are known problems with these drives,
THIS DRIVE MAY OR MAY NOT BE AFFECTED,
see the following web pages for details:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/207951en
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=632758
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Unknown
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 650) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 120) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x103b) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 099 088 006 - 160388856
3 Spin_Up_Time PO---- 094 088 000 - 0
4 Start_Stop_Count -O--CK 099 099 020 - 1288
5 Reallocated_Sector_Ct PO--CK 100 100 036 - 1290
7 Seek_Error_Rate POSR-- 069 060 030 - 77467799408
9 Power_On_Hours -O--CK 061 061 000 - 34392
10 Spin_Retry_Count PO--C- 100 100 097 - 0
12 Power_Cycle_Count -O--CK 100 100 020 - 988
184 End-to-End_Error -O--CK 100 100 099 - 0
187 Reported_Uncorrect -O--CK 001 001 000 - 183
188 Command_Timeout -O--CK 100 096 000 - 4295032839
189 High_Fly_Writes -O-RCK 100 100 000 - 0
190 Airflow_Temperature_Cel -O---K 067 054 045 - 33 (Min/Max 16/33)
194 Temperature_Celsius -O---K 033 046 000 - 33 (0 10 0 0 0)
195 Hardware_ECC_Recovered -O-RC- 024 016 000 - 160388856
197 Current_Pending_Sector -O--C- 100 100 000 - 757
198 Offline_Uncorrectable ----C- 100 100 000 - 757
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 GPL,SL R/O 1 Summary SMART error log
0x02 GPL,SL R/O 5 Comprehensive SMART error log
0x03 GPL,SL R/O 5 Ext. Comprehensive SMART error log
0x06 GPL,SL R/O 1 SMART self-test log
0x07 GPL,SL R/O 1 Extended self-test log
0x09 GPL,SL R/W 1 Selective self-test log
0x10 GPL,SL R/O 1 NCQ Command Error log
0x11 GPL,SL R/O 1 SATA Phy Event Counters
0x21 GPL,SL R/O 1 Write stream error log
0x22 GPL,SL R/O 1 Read stream error log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa1 GPL,SL VS 20 Device vendor specific log
0xa2 GPL VS 2248 Device vendor specific log
0xa8 GPL,SL VS 20 Device vendor specific log
0xa9 GPL,SL VS 1 Device vendor specific log
0xb0 GPL VS 2819 Device vendor specific log
0xbe-0xbf GPL VS 65535 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
Device Error Count: 261 (device log contains only the most recent 20 errors)
CR = Command Register
FEATR = Features Register
COUNT = Count (was: Sector Count) Register
LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8
LH = LBA High (was: Cylinder High) Register ] LBA
LM = LBA Mid (was: Cylinder Low) Register ] Register
LL = LBA Low (was: Sector Number) Register ]
DV = Device (was: Device/Head) Register
DC = Device Control Register
ER = Error register
ST = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 261 [0] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d e0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
a1 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:14.067 IDENTIFY PACKET DEVICE
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:14.067 IDENTIFY DEVICE
00 00 00 00 00 00 00 00 00 00 00 00 04 13:02:13.913 NOP [Abort
queued commands]
00 00 00 00 00 00 00 00 00 00 00 00 ff 13:02:13.587 NOP [Abort
queued commands]
a1 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.581 IDENTIFY PACKET DEVICE
Error 260 [19] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d e0 00
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:14.067 IDENTIFY DEVICE
00 00 00 00 00 00 00 00 00 00 00 00 04 13:02:13.913 NOP [Abort
queued commands]
00 00 00 00 00 00 00 00 00 00 00 00 ff 13:02:13.587 NOP [Abort
queued commands]
a1 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.581 IDENTIFY PACKET DEVICE
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.580 IDENTIFY DEVICE
Error 259 [18] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d e0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
a1 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.581 IDENTIFY PACKET DEVICE
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.580 IDENTIFY DEVICE
00 00 00 00 00 00 00 00 00 00 00 00 04 13:02:08.427 NOP [Abort
queued commands]
00 00 00 00 00 00 00 00 00 00 00 00 ff 13:02:08.107 NOP [Abort
queued commands]
a1 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.086 IDENTIFY PACKET DEVICE
Error 258 [17] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d e0 00
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.580 IDENTIFY DEVICE
00 00 00 00 00 00 00 00 00 00 00 00 04 13:02:08.427 NOP [Abort
queued commands]
00 00 00 00 00 00 00 00 00 00 00 00 ff 13:02:08.107 NOP [Abort
queued commands]
a1 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.086 IDENTIFY PACKET DEVICE
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.069 IDENTIFY DEVICE
Error 257 [16] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d e0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
a1 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.086 IDENTIFY PACKET DEVICE
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.069 IDENTIFY DEVICE
2f 00 00 00 01 00 00 00 00 00 10 a0 00 13:02:08.047 READ LOG EXT
60 00 00 00 10 00 00 81 00 05 ef 40 00 13:02:04.913 READ FPDMA QUEUED
60 00 00 00 08 00 00 83 00 04 9f 40 00 13:02:04.913 READ FPDMA QUEUED
Error 256 [15] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d e0 00
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
ec 00 00 00 00 00 00 00 00 00 00 a0 00 13:02:08.069 IDENTIFY DEVICE
2f 00 00 00 01 00 00 00 00 00 10 a0 00 13:02:08.047 READ LOG EXT
60 00 00 00 10 00 00 81 00 05 ef 40 00 13:02:04.913 READ FPDMA QUEUED
60 00 00 00 08 00 00 83 00 04 9f 40 00 13:02:04.913 READ FPDMA QUEUED
60 00 00 00 08 00 00 41 00 08 47 40 00 13:02:04.912 READ FPDMA QUEUED
Error 255 [14] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d e0 00 Device Fault; Error: ABRT
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
2f 00 00 00 01 00 00 00 00 00 10 a0 00 13:02:08.047 READ LOG EXT
60 00 00 00 10 00 00 81 00 05 ef 40 00 13:02:04.913 READ FPDMA QUEUED
60 00 00 00 08 00 00 83 00 04 9f 40 00 13:02:04.913 READ FPDMA QUEUED
60 00 00 00 08 00 00 41 00 08 47 40 00 13:02:04.912 READ FPDMA QUEUED
60 00 00 00 08 00 00 70 00 06 b7 40 00 13:02:04.909 READ FPDMA QUEUED
Error 254 [13] occurred at disk power-on lifetime: 34388 hours (1432
days + 20 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
04 -- 71 00 04 00 00 00 00 32 9d 40 00 Device Fault; Error: ABRT at
LBA = 0x0000329d = 12957
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 00 00 10 00 00 81 00 05 ef 40 00 13:02:04.913 READ FPDMA QUEUED
60 00 00 00 08 00 00 83 00 04 9f 40 00 13:02:04.913 READ FPDMA QUEUED
60 00 00 00 08 00 00 41 00 08 47 40 00 13:02:04.912 READ FPDMA QUEUED
60 00 00 00 08 00 00 70 00 06 b7 40 00 13:02:04.909 READ FPDMA QUEUED
60 00 00 00 08 00 00 7e 00 03 87 40 00 13:02:04.909 READ FPDMA QUEUED
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 34220
336941
# 2 Extended offline Completed: read failure 90% 32539
39315653
# 3 Extended offline Completed without error 00% 25079 -
# 4 Extended offline Interrupted (host reset) 00% 24385 -
# 5 Short offline Completed without error 00% 19163 -
# 6 Short offline Completed without error 00% 10742 -
# 7 Extended offline Completed without error 00% 79 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 522 (0x020a)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 33 Celsius
Power Cycle Min/Max Temperature: 16/33 Celsius
Lifetime Min/Max Temperature: 10/46 Celsius
Under/Over Temperature Limit Count: 0/235
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/ 0 Celsius
Min/Max Temperature Limit: 0/ 0 Celsius
Temperature History Size (Index): 128 (50)
Index Estimated Time Temperature Celsius
51 2015-02-16 20:02 30 ***********
... ..( 12 skipped). .. ***********
64 2015-02-16 20:15 30 ***********
65 2015-02-16 20:16 28 *********
66 2015-02-16 20:17 28 *********
67 2015-02-16 20:18 28 *********
68 2015-02-16 20:19 29 **********
... ..( 6 skipped). .. **********
75 2015-02-16 20:26 29 **********
76 2015-02-16 20:27 30 ***********
... ..( 9 skipped). .. ***********
86 2015-02-16 20:37 30 ***********
87 2015-02-16 20:38 31 ************
... ..( 40 skipped). .. ************
0 2015-02-16 21:19 31 ************
1 2015-02-16 21:20 32 *************
... ..( 38 skipped). .. *************
40 2015-02-16 21:59 32 *************
41 2015-02-16 22:00 33 **************
... ..( 8 skipped). .. **************
50 2015-02-16 22:09 33 **************
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
Device Statistics (GP Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x000a 2 14 Device-to-host register FISes sent due to a COMRESET
0x0001 2 0 Command failed due to ICRC error
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
# smartctl -x /dev/sdc
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.18.6-1-ARCH] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Black
Device Model: WDC WD5001AALS-00E3A0
Serial Number: WD-WCATR3121296
LU WWN Device Id: 5 0014ee 25a457f4e
Firmware Version: 05.01D05
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.6, 3.0 Gb/s
Local Time is: Mon Feb 16 22:09:29 2015 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Disabled
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 8700) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 103) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x3037) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0
3 Spin_Up_Time POS--K 179 174 021 - 4050
4 Start_Stop_Count -O--CK 100 100 000 - 963
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 068 068 000 - 23596
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 739
192 Power-Off_Retract_Count -O--CK 200 200 000 - 97
193 Load_Cycle_Count -O--CK 200 200 000 - 865
194 Temperature_Celsius -O---K 111 097 000 - 36
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 200 200 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 6 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa0-0xa7 GPL,SL VS 16 Device vendor specific log
0xa8-0xb5 GPL,SL VS 1 Device vendor specific log
0xb6 GPL VS 1 Device vendor specific log
0xb7 GPL,SL VS 1 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL VS 24 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 23424 -
# 2 Extended offline Completed without error 00% 21741 -
# 3 Extended offline Completed without error 00% 14306 -
# 4 Short offline Completed without error 00% 8409 -
# 5 Short offline Completed without error 00% 2 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 2
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 36 Celsius
Power Cycle Min/Max Temperature: 18/37 Celsius
Lifetime Min/Max Temperature: 36/50 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (252)
Index Estimated Time Temperature Celsius
253 2015-02-16 14:12 35 ****************
... ..( 6 skipped). .. ****************
260 2015-02-16 14:19 35 ****************
261 2015-02-16 14:20 36 *****************
... ..( 16 skipped). .. *****************
278 2015-02-16 14:37 36 *****************
279 2015-02-16 14:38 37 ******************
280 2015-02-16 14:39 36 *****************
... ..( 2 skipped). .. *****************
283 2015-02-16 14:42 36 *****************
284 2015-02-16 14:43 37 ******************
285 2015-02-16 14:44 36 *****************
... ..( 54 skipped). .. *****************
340 2015-02-16 15:39 36 *****************
341 2015-02-16 15:40 37 ******************
342 2015-02-16 15:41 36 *****************
... ..( 3 skipped). .. *****************
346 2015-02-16 15:45 36 *****************
347 2015-02-16 15:46 37 ******************
348 2015-02-16 15:47 36 *****************
... ..( 4 skipped). .. *****************
353 2015-02-16 15:52 36 *****************
354 2015-02-16 15:53 37 ******************
355 2015-02-16 15:54 36 *****************
... ..( 2 skipped). .. *****************
358 2015-02-16 15:57 36 *****************
359 2015-02-16 15:58 37 ******************
360 2015-02-16 15:59 36 *****************
... ..( 10 skipped). .. *****************
371 2015-02-16 16:10 36 *****************
372 2015-02-16 16:11 37 ******************
373 2015-02-16 16:12 36 *****************
... ..( 12 skipped). .. *****************
386 2015-02-16 16:25 36 *****************
387 2015-02-16 16:26 37 ******************
388 2015-02-16 16:27 36 *****************
... ..(145 skipped). .. *****************
56 2015-02-16 18:53 36 *****************
57 2015-02-16 18:54 35 ****************
... ..( 46 skipped). .. ****************
104 2015-02-16 19:41 35 ****************
105 2015-02-16 19:42 ? -
106 2015-02-16 19:43 18 -
107 2015-02-16 19:44 19 -
108 2015-02-16 19:45 20 *
109 2015-02-16 19:46 20 *
110 2015-02-16 19:47 21 **
111 2015-02-16 19:48 21 **
112 2015-02-16 19:49 22 ***
113 2015-02-16 19:50 22 ***
114 2015-02-16 19:51 23 ****
115 2015-02-16 19:52 24 *****
116 2015-02-16 19:53 25 ******
117 2015-02-16 19:54 25 ******
118 2015-02-16 19:55 25 ******
119 2015-02-16 19:56 26 *******
... ..( 3 skipped). .. *******
123 2015-02-16 20:00 26 *******
124 2015-02-16 20:01 27 ********
125 2015-02-16 20:02 27 ********
126 2015-02-16 20:03 27 ********
127 2015-02-16 20:04 28 *********
128 2015-02-16 20:05 28 *********
129 2015-02-16 20:06 29 **********
... ..( 4 skipped). .. **********
134 2015-02-16 20:11 29 **********
135 2015-02-16 20:12 30 ***********
... ..( 4 skipped). .. ***********
140 2015-02-16 20:17 30 ***********
141 2015-02-16 20:18 31 ************
142 2015-02-16 20:19 31 ************
143 2015-02-16 20:20 30 ***********
144 2015-02-16 20:21 31 ************
... ..( 3 skipped). .. ************
148 2015-02-16 20:25 31 ************
149 2015-02-16 20:26 32 *************
... ..( 18 skipped). .. *************
168 2015-02-16 20:45 32 *************
169 2015-02-16 20:46 33 **************
... ..( 16 skipped). .. **************
186 2015-02-16 21:03 33 **************
187 2015-02-16 21:04 34 ***************
... ..( 2 skipped). .. ***************
190 2015-02-16 21:07 34 ***************
191 2015-02-16 21:08 35 ****************
192 2015-02-16 21:09 34 ***************
193 2015-02-16 21:10 35 ****************
... ..( 31 skipped). .. ****************
225 2015-02-16 21:42 35 ****************
226 2015-02-16 21:43 36 *****************
227 2015-02-16 21:44 36 *****************
228 2015-02-16 21:45 35 ****************
229 2015-02-16 21:46 34 ***************
230 2015-02-16 21:47 33 **************
231 2015-02-16 21:48 33 **************
232 2015-02-16 21:49 ? -
233 2015-02-16 21:50 33 **************
234 2015-02-16 21:51 32 *************
235 2015-02-16 21:52 32 *************
236 2015-02-16 21:53 33 **************
... ..( 8 skipped). .. **************
245 2015-02-16 22:02 33 **************
246 2015-02-16 22:03 34 ***************
... ..( 3 skipped). .. ***************
250 2015-02-16 22:07 34 ***************
251 2015-02-16 22:08 35 ****************
252 2015-02-16 22:09 35 ****************
SCT Error Recovery Control command not supported
Device Statistics (GP Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x000a 2 9 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x8000 4 14489 Vendor specific
# smartctl -x /dev/sdd
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.18.6-1-ARCH] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: SAMSUNG SpinPoint F3
Device Model: SAMSUNG HD103SJ
Serial Number: S246J9KB924045
LU WWN Device Id: 5 0024e9 2062a5827
Firmware Version: 1AJ10001
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 6
SATA Version is: SATA 2.6, 3.0 Gb/s
Local Time is: Mon Feb 16 22:10:00 2015 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Disabled
APM feature is: Disabled
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 9360) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 156) minutes.
SCT capabilities: (0x003f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 100 100 051 - 1
2 Throughput_Performance -OS--K 055 050 000 - 8667
3 Spin_Up_Time PO---K 069 067 025 - 9424
4 Start_Stop_Count -O--CK 100 100 000 - 589
5 Reallocated_Sector_Ct PO--CK 252 252 010 - 0
7 Seek_Error_Rate -OSR-K 252 252 051 - 0
8 Seek_Time_Performance --S--K 252 252 015 - 0
9 Power_On_Hours -O--CK 100 100 000 - 15097
10 Spin_Retry_Count -O--CK 252 252 051 - 0
11 Calibration_Retry_Count -O--CK 252 252 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 491
191 G-Sense_Error_Rate -O---K 100 100 000 - 3
192 Power-Off_Retract_Count -O---K 252 252 000 - 0
194 Temperature_Celsius -O---- 064 055 000 - 34 (Min/Max 11/45)
195 Hardware_ECC_Recovered -O-RCK 100 100 000 - 0
196 Reallocated_Event_Count -O--CK 252 252 000 - 0
197 Current_Pending_Sector -O--CK 252 252 000 - 0
198 Offline_Uncorrectable ----CK 252 252 000 - 0
199 UDMA_CRC_Error_Count -OS-CK 200 200 000 - 0
200 Multi_Zone_Error_Rate -O-R-K 100 100 000 - 30
223 Load_Retry_Count -O--CK 252 252 000 - 0
225 Load_Cycle_Count -O--CK 100 100 000 - 591
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 2 Comprehensive SMART error log
0x03 GPL R/O 2 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 2 Extended self-test log
0x08 GPL R/O 2 Power Conditions log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (2 sectors)
No Errors Logged
SMART Extended Self-test Log Version: 1 (2 sectors)
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 14928 -
# 2 Extended offline Completed without error 00% 13234 -
# 3 Extended offline Completed without error 00% 5808 -
# 4 Extended offline Interrupted (host reset) 70% 5136 -
SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has
ever been run
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Completed [00% left] (0-65535)
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 2
SCT Version (vendor specific): 256 (0x0100)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 34 Celsius
Power Cycle Min/Max Temperature: 19/34 Celsius
Lifetime Min/Max Temperature: 14/64 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 5 minutes
Temperature Logging Interval: 5 minutes
Min/Max recommended Temperature: -5/80 Celsius
Min/Max Temperature Limit: -10/85 Celsius
Temperature History Size (Index): 128 (7)
Index Estimated Time Temperature Celsius
8 2015-02-16 11:35 35 ****************
9 2015-02-16 11:40 35 ****************
10 2015-02-16 11:45 34 ***************
... ..( 22 skipped). .. ***************
33 2015-02-16 13:40 34 ***************
34 2015-02-16 13:45 35 ****************
35 2015-02-16 13:50 34 ***************
... ..( 22 skipped). .. ***************
58 2015-02-16 15:45 34 ***************
59 2015-02-16 15:50 33 **************
60 2015-02-16 15:55 33 **************
61 2015-02-16 16:00 34 ***************
62 2015-02-16 16:05 33 **************
63 2015-02-16 16:10 34 ***************
64 2015-02-16 16:15 33 **************
... ..( 6 skipped). .. **************
71 2015-02-16 16:50 33 **************
72 2015-02-16 16:55 34 ***************
73 2015-02-16 17:00 33 **************
74 2015-02-16 17:05 33 **************
75 2015-02-16 17:10 33 **************
76 2015-02-16 17:15 34 ***************
77 2015-02-16 17:20 33 **************
... ..( 7 skipped). .. **************
85 2015-02-16 18:00 33 **************
86 2015-02-16 18:05 32 *************
87 2015-02-16 18:10 33 **************
88 2015-02-16 18:15 33 **************
89 2015-02-16 18:20 32 *************
90 2015-02-16 18:25 19 -
91 2015-02-16 18:30 22 ***
92 2015-02-16 18:35 24 *****
93 2015-02-16 18:40 25 ******
94 2015-02-16 18:45 26 *******
95 2015-02-16 18:50 27 ********
96 2015-02-16 18:55 28 *********
97 2015-02-16 19:00 28 *********
98 2015-02-16 19:05 29 **********
99 2015-02-16 19:10 29 **********
100 2015-02-16 19:15 30 ***********
... ..( 3 skipped). .. ***********
104 2015-02-16 19:35 30 ***********
105 2015-02-16 19:40 31 ************
... ..( 5 skipped). .. ************
111 2015-02-16 20:10 31 ************
112 2015-02-16 20:15 32 *************
113 2015-02-16 20:20 30 ***********
114 2015-02-16 20:25 30 ***********
115 2015-02-16 20:30 31 ************
116 2015-02-16 20:35 31 ************
117 2015-02-16 20:40 32 *************
118 2015-02-16 20:45 32 *************
119 2015-02-16 20:50 32 *************
120 2015-02-16 20:55 33 **************
... ..( 7 skipped). .. **************
0 2015-02-16 21:35 33 **************
1 2015-02-16 21:40 34 ***************
... ..( 5 skipped). .. ***************
7 2015-02-16 22:10 34 ***************
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
Device Statistics (GP Log 0x04) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 4 0 Command failed due to ICRC error
0x0002 4 0 R_ERR response for data FIS
0x0003 4 0 R_ERR response for device-to-host data FIS
0x0004 4 0 R_ERR response for host-to-device data FIS
0x0005 4 0 R_ERR response for non-data FIS
0x0006 4 0 R_ERR response for device-to-host non-data FIS
0x0007 4 0 R_ERR response for host-to-device non-data FIS
0x0008 4 0 Device-to-host non-data FIS retries
0x0009 4 16 Transition from drive PhyRdy to drive PhyNRdy
0x000a 4 14 Device-to-host register FISes sent due to a COMRESET
0x000b 4 0 CRC errors within host-to-device FIS
0x000d 4 0 Non-CRC errors within host-to-device FIS
0x000f 4 0 R_ERR response for host-to-device data FIS, CRC
0x0010 4 0 R_ERR response for host-to-device data FIS, non-CRC
0x0012 4 0 R_ERR response for host-to-device non-data FIS, CRC
0x0013 4 0 R_ERR response for host-to-device non-data FIS, non-CRC
0x8e00 4 0 Vendor specific
0x8e01 4 0 Vendor specific
0x8e02 4 0 Vendor specific
0x8e03 4 0 Vendor specific
0x8e04 4 0 Vendor specific
0x8e05 4 0 Vendor specific
0x8e06 4 0 Vendor specific
0x8e07 4 0 Vendor specific
0x8e08 4 0 Vendor specific
0x8e09 4 0 Vendor specific
0x8e0a 4 0 Vendor specific
0x8e0b 4 0 Vendor specific
0x8e0c 4 0 Vendor specific
0x8e0d 4 0 Vendor specific
0x8e0e 4 0 Vendor specific
0x8e0f 4 0 Vendor specific
0x8e10 4 0 Vendor specific
0x8e11 4 0 Vendor specific
No /dev/sde anymore...
> Did you read about and fix any timeout mismatch issue?
I did yes but not sure to understand how to set up 120 second timeout
in sysfs...
The following commands is still valid without 1 of my drives?
mdadm --stop /dev/md126
mdadm --assemble --force --verbose --run /dev/md126 /dev/sd[bcd]3
> Also, your device names have changed. You *must* keep track of which
> one is which "RaidDevice". Specifically, what was "sde" now appears to
> be "sdd". Did you reboot? You know that device names are not
> guaranteed to be consistent from one boot to the next, I hope. Show an
> excerpt from "ls -l /dev/disk/by-id/" with your next report so we know
> which drive serial number has which name.
Now you said it, I'm felling stupid... Yes, I reboot every day and I
don't have trace of previous "mapping" (id -> name).
# ls -l /dev/disk/by-id
total 0
lrwxrwxrwx 1 root root 9 Feb 16 21:16
ata-SAMSUNG_HD103SJ_S246J9KB924045 -> ../../sdd
lrwxrwxrwx 1 root root 10 Feb 16 21:16
ata-SAMSUNG_HD103SJ_S246J9KB924045-part1 -> ../../sdd1
lrwxrwxrwx 1 root root 10 Feb 16 21:16
ata-SAMSUNG_HD103SJ_S246J9KB924045-part2 -> ../../sdd2
lrwxrwxrwx 1 root root 10 Feb 16 21:16
ata-SAMSUNG_HD103SJ_S246J9KB924045-part3 -> ../../sdd3
lrwxrwxrwx 1 root root 10 Feb 16 21:16
ata-SAMSUNG_HD103SJ_S246J9KB924045-part4 -> ../../sdd4
lrwxrwxrwx 1 root root 9 Feb 16 21:16 ata-ST3500320AS_9QM8H48M -> ../../sdb
lrwxrwxrwx 1 root root 10 Feb 16 21:16 ata-ST3500320AS_9QM8H48M-part1
-> ../../sdb1
lrwxrwxrwx 1 root root 10 Feb 16 21:16 ata-ST3500320AS_9QM8H48M-part2
-> ../../sdb2
lrwxrwxrwx 1 root root 10 Feb 16 21:16 ata-ST3500320AS_9QM8H48M-part3
-> ../../sdb3
lrwxrwxrwx 1 root root 9 Feb 16 21:16
ata-WDC_WD5001AALS-00E3A0_WD-WCATR3121296 -> ../../sdc
lrwxrwxrwx 1 root root 10 Feb 16 21:16
ata-WDC_WD5001AALS-00E3A0_WD-WCATR3121296-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Feb 16 21:16
ata-WDC_WD5001AALS-00E3A0_WD-WCATR3121296-part2 -> ../../sdc2
lrwxrwxrwx 1 root root 10 Feb 16 21:16
ata-WDC_WD5001AALS-00E3A0_WD-WCATR3121296-part3 -> ../../sdc3
lrwxrwxrwx 1 root root 11 Feb 16 21:17
md-uuid-6408f077:eefbe269:c173be8e:b17eb4df -> ../../md127
lrwxrwxrwx 1 root root 9 Feb 16 21:16 wwn-0x5000c5001099d533 -> ../../sdb
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x5000c5001099d533-part1 ->
../../sdb1
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x5000c5001099d533-part2 ->
../../sdb2
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x5000c5001099d533-part3 ->
../../sdb3
lrwxrwxrwx 1 root root 9 Feb 16 21:16 wwn-0x50014ee25a457f4e -> ../../sdc
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x50014ee25a457f4e-part1 ->
../../sdc1
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x50014ee25a457f4e-part2 ->
../../sdc2
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x50014ee25a457f4e-part3 ->
../../sdc3
lrwxrwxrwx 1 root root 9 Feb 16 21:16 wwn-0x50024e92062a5827 -> ../../sdd
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x50024e92062a5827-part1 ->
../../sdd1
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x50024e92062a5827-part2 ->
../../sdd2
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x50024e92062a5827-part3 ->
../../sdd3
lrwxrwxrwx 1 root root 10 Feb 16 21:16 wwn-0x50024e92062a5827-part4 ->
../../sdd4
Thanks,
Dush
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Jes Sorensen @ 2015-02-16 22:49 UTC (permalink / raw)
To: NeilBrown; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <wrfjpp99pp9d.fsf@redhat.com>
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> NeilBrown <neilb@suse.de> writes:
>> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
>> wrote:
>>
>>> Dear All,
>>> Any updates on this issue.
>>
>> Probably the same as:
>>
>> http://marc.info/?l=linux-raid&m=142283560704091&w=2
>
> Hi Neil,
>
> I ran some tests on this one against the latest Linus' tree as of today
> (1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
> your pending 3.20 patches.
>
> I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
> run fio on top of a device while it is resyncing and I fail one of the
> devices.
Since Manibalan mentioned this issue wasn't present in earlier kernels,
I started trying to track down what change caused it.
So far I have been able to reproduce the hang as far back as 3.10.
Cheers,
Jes
^ permalink raw reply
* Re: What are mdadm maintainers to do? (was: desktop disk's error recovery timeouts)
From: NeilBrown @ 2015-02-16 23:49 UTC (permalink / raw)
To: Chris; +Cc: linux-raid
In-Reply-To: <loom.20150216T183419-387@post.gmane.org>
[-- Attachment #1: Type: text/plain, Size: 1067 bytes --]
On Mon, 16 Feb 2015 17:48:50 +0000 (UTC) Chris <email.bug@arcor.de> wrote:
>
> Thank you for the additional information, it calls for action.
>
>
> OK, calling for a solution to stop desktop drives from causing data loss and
> affecting the mdadm reputation:
>
>
> I gather that mdadm could ship with one additional udev rule that calls a
> script to check/set scterc, or falls back to increasing the system timout.
>
> Phil, you mentioned having posted such a script, could you prepare it for
> addition to the mdadm package?
>
>
> Would maintainers be ok with adding such a udev rule and script to the package?
"maintainers" ? Plural? That would be nice.
Unfortunately there is just the one singular me....
There are certainly other contributors who
- answer questions on the list
- provide bug reports
- provide bits of code
and I am very thankful to them. But I haven't found a likely co-maintainer
yet :-(
I'm certainly happy to consider and concrete proposal. The more concrete,
the better.
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Chris @ 2015-02-17 7:52 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <20150217104906.62d36c62@notabene.brown>
NeilBrown <neilb <at> suse.de> writes:
> "maintainers" ? Plural? That would be nice.
> Unfortunately there is just the one singular me....
Yes, as Weedy said, I also refered to distro package maintainers.
If we can come up here with an udev rule and a script to call, then upstream
(you) could include this, and distro maintainers could make smartctl a
suggested or recommended package of the mdadm package.
I certainly have not understood the whole topic yet,
what I just got is, that the script should do something like
the following, and I found some implementation below.
Evererybody please answer with improved versions if you can.
if smartctl tool is available
if scterc is disabled
/usr/sbin/smartctl -l scterc,70,70 ${DEVNAME}
else
if screrc is not available
echo 180 >/sys/block/${DEVNAME}/device/timeout
Found an older implementation that "seems to work fine":
http://article.gmane.org/gmane.linux.raid/44566
>
> contents of udev rule:
> ACTION=="add", SUBSYSTEM=="block", KERNEL=="[sh]d[a-z]",
RUN+="/usr/local/bin/settimeout"
>
>
> contents of /usr/local/bin/settimeout:
> #!/bin/bash
>
> [ "${ACTION}" == "add" ] && {
> /usr/sbin/smartctl -l scterc,70,70 ${DEVNAME} || echo 180 >
/sys/${DEVPATH}/device/timeout
> }
>
> I guess, what is missing, is to connect the HDDs
> with a specific "mdadm" event, instead of running
> for each HDD.
> I'm not sure if this is already possible, since
> some "udev" rules for "md" are already existing.
Let's get this disaster prevention into mdadm, even if just as important
reference experience for solving a more general kernel timeout mismatch
problem "symptom of a more generic issue".
http://article.gmane.org/gmane.linux.raid/44557
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Mikael Abrahamsson @ 2015-02-17 8:48 UTC (permalink / raw)
To: Chris; +Cc: linux-raid
In-Reply-To: <loom.20150217T080345-764@post.gmane.org>
On Tue, 17 Feb 2015, Chris wrote:
> Evererybody please answer with improved versions if you can.
>
> if smartctl tool is available
> if scterc is disabled
> /usr/sbin/smartctl -l scterc,70,70 ${DEVNAME}
> else
> if screrc is not available
> echo 180 >/sys/block/${DEVNAME}/device/timeout
>
> Found an older implementation that "seems to work fine":
Hi,
Generally I like this idea, and I agree that this would be a good idea,
but if I was running raid0 or linear, I might not want scterc to be
enabled.
Also, what would the harm be to always bump the timeout to 180 seconds?
Yes, drives would take longer to be kicked out in case of errors, but if
we're confident in scterc working, wouldn't we want to turn down the
timeout to 10-15 seconds then?
Personally I turn on scterc if available and turn up the timeout to 180
seconds, always, regardless what drives I'm running. I'd rather wait
longer for a drive to be considered dead, than to have drives being kicked
due to some hiccup in the system (controller or drive reset) that might
rectify itself.
So I would suggest turning on scterc and turning up the timeout to 180
seconds as soon as mdadm is installed. This is the best tradeoff I can
come up with between stability and fast drive-dead-detection time.
Here on the list I see people all the time coming in with multiple drives
kicked due to controller resets and other intermittent flukes, I never see
people coming in complaining that it took 30 seconds to detect a drive
error. I doubt there'd be much complaint for 180 seconds. If someone needs
faster detect times then my opinion is that they are in the category who
can be expected to tune this value to their application. 180 seconds works
best for the "larger crowd" using mdadm.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Chris @ 2015-02-17 10:37 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <alpine.DEB.2.02.1502170940010.4007@uplift.swm.pp.se>
Mikael Abrahamsson <swmike <at> swm.pp.se> writes:
> if I was running raid0 or linear, I might not want scterc to be
> enabled.
Good Point.
> Also, what would the harm be to always bump the timeout to 180 seconds?
I don't know why the driver authors chose that linux default,
but the todo with both your points:
if the appearind device is an md member device (mdadm examine?)
if smartctl tool is available
if scterc is disabled in cotaining ${HDD_DEV} AND added device is not
raid0/linear
/usr/sbin/smartctl -l scterc,70,70 ${HDD_DEV}
echo 180 >/sys/block/${HDD_DEV}/device/timeout
^ permalink raw reply
* Re: re-add POLICY
From: Chris @ 2015-02-17 15:09 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <loom.20150216T124230-883@post.gmane.org>
> NeilBrown <neilb <at> suse.de> writes:
>
> > If it doesn't, then maybe you need "POLICY action=spare".
>
> OK, I will test this when the notebook is back in the house.
I could test it on another system.
Without adding a bitmap, it required configring
POLICY domain=default action=spare
and calling
mdadm --udev-rules
but then, after removing and inserting sdc again, only two out of six md
partitions got synced.
To see if there is something wrong, I then added the sdc1 md0 member
manually, and it synced without failure.
So I can't tell why the other partitions did not sync atomatically.
Some of the unsynced partition types are 83 (md0 member), but others
are FD (md7 member) like the automatically synced ones.
linux 3.2.0
mdadm v3.2.5
md7 : active raid1 sdc6[3] sda8[2]
14327680 blocks super 1.2 [3/2] [UU_]
bitmap: 1/1 pages [4KB], 65536KB chunk
md3 : active raid1 sdc8[4] sda10[3]
307011392 blocks super 1.2 [3/2] [UU_]
bitmap: 3/3 pages [12KB], 65536KB chunk
md6 : active raid1 sda7[2]
8695680 blocks super 1.2 [3/1] [_U_]
md1 : active raid1 sda6[3](W) sdb2[1]
19513216 blocks super 1.2 [4/2] [_UU_]
md2 : active raid1 sda9[3](W) sdb3[0]
97590144 blocks super 1.2 [4/2] [U_U_]
md0 : active raid1 sdc1[4] sda5[2](W) sdb1[1]
340672 blocks super 1.2 [4/3] [UUU_]
A partition that did not sync automatically:
/dev/sdc7:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 7a5847cd:be0e8510:8e170bf5:5d40143f
Name : name:2 (local to host name)
Creation Time : Sun Dec 2 21:40:58 2012
Raid Level : raid1
Raid Devices : 4
Avail Dev Size : 195187135 (93.07 GiB 99.94 GB)
Array Size : 97590144 (93.07 GiB 99.93 GB)
Used Dev Size : 195180288 (93.07 GiB 99.93 GB)
Data Offset : 131072 sectors
Super Offset : 8 sectors
State : clean
Device UUID : b1a97d12:965e3d08:059acefb:6ac5b7e3
Update Time : Wed Dec 3 11:23:26 2014
Checksum : ac0ce511 - correct
Events : 382479
Device Role : Active device 1
Array State : AAA. ('A' == active, '.' == missing)
And a corresponding partition that is part of the running array:
/dev/sda9:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 7a5847cd:be0e8510:8e170bf5:5d40143f
Name : name:2 (local to host name)
Creation Time : Sun Dec 2 21:40:58 2012
Raid Level : raid1
Raid Devices : 4
Avail Dev Size : 195182592 (93.07 GiB 99.93 GB)
Array Size : 97590144 (93.07 GiB 99.93 GB)
Used Dev Size : 195180288 (93.07 GiB 99.93 GB)
Data Offset : 131072 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4c191282:80769896:378abe34:aeb01b8d
Flags : write-mostly
Update Time : Mon Feb 16 17:41:54 2015
Checksum : 8cb4794c - correct
Events : 384989
Device Role : Active device 2
Array State : A.A. ('A' == active, '.' == missing)
BTW looking at this data now, it seems to me the superblocks almost support
the clean re-sync / conflict detection I was trying to explain.
a) The removed device 1 does not claim that a member
in the running array (0 and 2) has failed (AAA.)
b) The Events count of device 1 is lower than in the running array.
c) The running array/superblock does not seem to keep
a reference of the Event count when device 1 failed, for additional
security that it has not ben started separately.
But b) and c) may not even be necessary, as starting device 1 separately
would make device 1 claim that 0 and 2 have failed, right?
Regards,
Chris
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Chris Murphy @ 2015-02-17 19:33 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <loom.20150217T080345-764@post.gmane.org>
It's not just mdadm. It likewise affects Btrfs, ZFS, and LVM.
Also, there's a lack of granularity with linux command timer and SCT
ERC applying only to the entire block device, not partitions. So
there's a problem for mixed use cases. For example, two drives, each
with two partitions. sda1 and sdb1 are raid0, and sda2 and sdb2 are
raid1. What's the proper configuration for SCT ERC and the SCSI
command timer?
*shrug* I don't think the automatic udev configuration idea is fail
safe. It sounds too easy for it to automatically cause a
misconfiguration. And it also doesn't at all solve the problem that
there's next to no error reporting to user space. smartd does, but
it's narrow in scope and entirely defers to the hard drive's
self-assessment. There's all sorts of problems that aren't in the
domain of SMART that get reported in dmesg, but there's no method for
gnome-shell or KDE or any DE or even send an email to a sysadmin, as
an early warning. Instead, all too often it's "WTF XFS just corrupted
itself!" meanwhile the real problem has been happening for a week,
dmesg/journal is full of errors indicating the nature of those
problems, but nothing bothered to inform a human being until the file
system face planted.
Chris Murphy
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Adam Goryachev @ 2015-02-17 22:47 UTC (permalink / raw)
To: Chris Murphy, linux-raid
In-Reply-To: <CAJCQCtQvoB0yPPc=RXPPyJhpaFwH0pQTRCRaR_ycTbzHrD6rLw@mail.gmail.com>
On 18/02/15 06:33, Chris Murphy wrote:
> It's not just mdadm. It likewise affects Btrfs, ZFS, and LVM.
>
> Also, there's a lack of granularity with linux command timer and SCT
> ERC applying only to the entire block device, not partitions. So
> there's a problem for mixed use cases. For example, two drives, each
> with two partitions. sda1 and sdb1 are raid0, and sda2 and sdb2 are
> raid1. What's the proper configuration for SCT ERC and the SCSI
> command timer?
Umm, actually I don't know enough to disagree, but I'll ask some
questions which probably shows both the assumptions I've made, and might
help others understand the issue better.
If we enable SCT ERC on every drive that supports it, and we are using
the drive (only) in a RAID0/linear array then what is the downside? As I
understand it, the drive will no longer try for > 120sec to recover the
data stored in the "bad" sector, and instead return an unreadable error
message in a short amount of time (well below 30 seconds) which means
the driver will be able to return a read error to the application (or FS
or MD) and the system as a whole will carry on. If we didn't enable SCT
ERC, then the entire drive would vanish, (because the timeout wasn't
changed for the driver) and the current read and every future read/write
will all fail, and the system will probably crash (well, depending on
the application, FS layout, etc).
So, IMHO, it seems that by default, every SCT ERC capable drive should
have this enabled by default. As a part of error recovery (ie, crap that
really important data stored on those few unreadable sectors) the user
could manually disable SCT ERC and re-attempt to request the data from
the drive (eg, during dd_rescue or similar).
Secondly, changing the timeout for those drives that don't support SCT
ERC, again, it is fairly similar to above, we get the error from the
drive before the timeout, except we will avoid the only possible
downside above (failing to read a very unlikely but possible to read
sector). Again, we will avoid dropping the entire drive, even if all
operations on this drive will stop for a longer period of time, it is
probably better than stopping permanently.
So, IMHO, every non SCT ERC capable drive should have the timeout
extended to 120s/180s or whatever the appropriate time is that (most)
drives will respond within. Leaving only the most extremely brain dead
drives which we simply ridicule on the list and anywhere and everywhere
possible to ensure nobody will ever buy them (or the manufacturer will
fix the problems).
Of course, quite possible I've totally over simplified this, and don't
understand the other repercussions?
> *shrug* I don't think the automatic udev configuration idea is fail
> safe. It sounds too easy for it to automatically cause a
> misconfiguration. And it also doesn't at all solve the problem that
> there's next to no error reporting to user space. smartd does, but
> it's narrow in scope and entirely defers to the hard drive's
> self-assessment. There's all sorts of problems that aren't in the
> domain of SMART that get reported in dmesg, but there's no method for
> gnome-shell or KDE or any DE or even send an email to a sysadmin, as
> an early warning. Instead, all too often it's "WTF XFS just corrupted
> itself!" meanwhile the real problem has been happening for a week,
> dmesg/journal is full of errors indicating the nature of those
> problems, but nothing bothered to inform a human being until the file
> system face planted.
Just because the solution doesn't solve the entire problem, it does
solve a part of the problem, so IMHO, better to solve this part of the
problem, and then discuss/try to find a solution to the rest of the
problem. Unless you have a suggestion which can solve both parts of the
problem? I suppose that a "good" sysadmin should install some sort of
log monitoring software which will alert them to issues, whether that is
via some desktop application/popup or email or something else. The
problem is that most of these issues come from "home" users who will
never setup anything like "log file monitoring" or raid scrubs, or
anything else, so if we do decide upon a generic solution that will work
for almost everybody, then we will still need to rely on the distro
maintainers to implement the solution.
PS, I suppose this is one of the "hide the gory details that nobody
understands" balancing with "provide the information to the user so they
can do something about it". One more generic consideration would be to
have the kernel identify which messages are purely informational/debug
and which are errors. Normal syslog has support for many different
levels, but AFAIK, all kernel messages end up in the same basket.
eg (plugging in and removing a USB drive generated the following log
entries as seen from "dmesg":
[614977.802828] usb 3-3: new high-speed USB device number 5 using xhci_hcd
[614977.822724] usb 3-3: New USB device found, idVendor=0951, idProduct=1665
[614977.822729] usb 3-3: New USB device strings: Mfr=1, Product=2,
SerialNumber=3
[614977.822732] usb 3-3: Product: DataTraveler 2.0
[614977.822735] usb 3-3: Manufacturer: Kingston
[614977.822737] usb 3-3: SerialNumber: 60A44C413CCBFE40AB4FFB3E
[614977.822899] usb 3-3: ep 0x81 - rounding interval to 128 microframes,
ep desc says 255 microframes
[614977.822905] usb 3-3: ep 0x2 - rounding interval to 128 microframes,
ep desc says 255 microframes
[614977.836547] usb-storage 3-3:1.0: USB Mass Storage device detected
[614977.836734] scsi6 : usb-storage 3-3:1.0
[614977.836819] usbcore: registered new interface driver usb-storage
[614978.854080] scsi 6:0:0:0: Direct-Access Kingston DataTraveler
2.0 1.00 PQ: 0 ANSI: 4
[614978.854493] sd 6:0:0:0: Attached scsi generic sg2 type 0
[614978.854658] sd 6:0:0:0: [sdb] 15131636 512-byte logical blocks:
(7.74 GB/7.21 GiB)
[614978.854884] sd 6:0:0:0: [sdb] Write Protect is off
[614978.854888] sd 6:0:0:0: [sdb] Mode Sense: 45 00 00 00
[614978.855085] sd 6:0:0:0: [sdb] Write cache: disabled, read cache:
enabled, doesn't support DPO or FUA
[614978.860015] sdb: sdb1
[614978.860864] sd 6:0:0:0: [sdb] Attached SCSI removable disk
[614979.061474] FAT-fs (sdb1): Volume was not properly unmounted. Some
data may be corrupt. Please run fsck.
[615347.862058] usb 3-3: reset high-speed USB device number 5 using xhci_hcd
[615347.862111] usb 3-3: Device not responding to set address.
[615348.065856] usb 3-3: Device not responding to set address.
[615348.269944] usb 3-3: device not accepting address 5, error -71
[615348.326429] usb 3-3: USB disconnect, device number 5
[615348.334730] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called
with disabled ep ffff88011b1b2600
[615348.334744] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called
with disabled ep ffff88011b1b2640
Of the above, I would suggest most of that is "info" while the following
lines might be warnings:
[614979.061474] FAT-fs (sdb1): Volume was not properly unmounted. Some
data may be corrupt. Please run fsck.
These might be error or critical:
[615347.862058] usb 3-3: reset high-speed USB device number 5 using xhci_hcd
[615347.862111] usb 3-3: Device not responding to set address.
[615348.065856] usb 3-3: Device not responding to set address.
[615348.269944] usb 3-3: device not accepting address 5, error -71
Of course, this will rely on every driver maintainer to make a decision
on just how important each line that they log may be.
Just my thoughts, hopefully it will be useful.
Regards,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au
^ permalink raw reply
* Re: What are mdadm maintainers to do? (error recovery redundancy/data loss)
From: Chris @ 2015-02-17 23:33 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAJCQCtQvoB0yPPc=RXPPyJhpaFwH0pQTRCRaR_ycTbzHrD6rLw@mail.gmail.com>
Chris Murphy writes:
>
> It's not just mdadm. It likewise affects Btrfs, ZFS, and LVM.
Do they have own timouts, or rely on the kernel?
Maybe the kernel could read the SCTERT value from the drives (in lieu of
some better retry timout information, and set the controller timout a little
greater than that, or very large if SCTERT is disabled/not available.
> sda1 and sdb1 are raid0, and sda2 and sdb2 are
> raid1. What's the proper configuration for SCT ERC and the SCSI
> command timer?
guessing...
For SCTERT disabled drives:
A compromise may be to stay with the linux default controller timout, it's
30s, and set the drives SCTERT below 30s (maybe 27s), to avoid losing
redundancy and risking data loss *AND* allow more of the available time for ERC.
For longer error correcting attempts (and just as long i/o controller
blocking!) the contoller timout could be set to 180s, and SCTERT to 175s?
BUT: If I chose to use a raid0 alongside a redundant raid I already
explicitly decided to take all data loss the hardware throws at me. So I
don't think it makes much of a difference if ERC times out after <30 secs or
180s, its just more or less errors belonging to me.
For SCTERC enabled drives:
30s and 7s seems ok?
> *shrug* I don't think the automatic udev configuration idea is fail
> safe. It sounds too easy for it to automatically cause a
> misconfiguration.
A matching timeout configuration prevents that unavoidable unrecoverable
read error take down the redundancy for sure, and cause high risk of data
loss during rebuild.
It does fix a misconfiguration, however could possibly set SCTERT just below
the (30s) controler timout, to reduce the impact of SCTERT (e.g make use of
the small chance of error correction succceding a couple of seconds later).
Given the longer SCTERT timout does not lead to subseqent read error timouts
piling up.
> And it also doesn't at all solve the problem that
> there's next to no error reporting to user space.
That is correct, but rather not related to the importance to fix the timout
mismatch and reduce the risk, is it? The settings do solve unecessary loss
of redundancy on read errors that are sure to occur, unnecessary resyncing,
and high risk of data loss during all that.
^ permalink raw reply
* Re: md_raid5 using 100% CPU and hang with status resync=PENDING, if a drive is removed during initialization
From: Jes Sorensen @ 2015-02-18 0:03 UTC (permalink / raw)
To: NeilBrown; +Cc: Manibalan P, Pasi Kärkkäinen, linux-raid
In-Reply-To: <wrfjd259o4iy.fsf@redhat.com>
Jes Sorensen <Jes.Sorensen@redhat.com> writes:
> Jes Sorensen <Jes.Sorensen@redhat.com> writes:
>> NeilBrown <neilb@suse.de> writes:
>>> On Mon, 2 Feb 2015 07:10:14 +0000 Manibalan P <pmanibalan@amiindia.co.in>
>>> wrote:
>>>
>>>> Dear All,
>>>> Any updates on this issue.
>>>
>>> Probably the same as:
>>>
>>> http://marc.info/?l=linux-raid&m=142283560704091&w=2
>>
>> Hi Neil,
>>
>> I ran some tests on this one against the latest Linus' tree as of today
>> (1fa185ebcbcefdc5229c783450c9f0439a69f0c1) which I believe includes all
>> your pending 3.20 patches.
>>
>> I am able to reproduce Manibalan's hangs on a system with 4 SSDs if I
>> run fio on top of a device while it is resyncing and I fail one of the
>> devices.
>
> Since Manibalan mentioned this issue wasn't present in earlier kernels,
> I started trying to track down what change caused it.
>
> So far I have been able to reproduce the hang as far back as 3.10.
After a lot of bisecting I finally traced the issue back to this commit:
a7854487cd7128a30a7f4f5259de9f67d5efb95f is the first bad commit
commit a7854487cd7128a30a7f4f5259de9f67d5efb95f
Author: Alexander Lyakas <alex.bolshoy@gmail.com>
Date: Thu Oct 11 13:50:12 2012 +1100
md: When RAID5 is dirty, force reconstruct-write instead of read-modify-write.
Signed-off-by: Alex Lyakas <alex@zadarastorage.com>
Suggested-by: Yair Hershko <yair@zadarastorage.com>
Signed-off-by: NeilBrown <neilb@suse.de>
If I revert that one I cannot reproduce the hang, applying it reproduces
the hang consistently.
Cheers,
Jes
^ permalink raw reply
* [PATCH] Revert "md: When RAID5 is dirty, force reconstruct-write instead of read-modify-write."
From: Jes.Sorensen @ 2015-02-18 0:14 UTC (permalink / raw)
To: neilb; +Cc: linux-raid, Jes Sorensen
From: Jes Sorensen <Jes.Sorensen@redhat.com>
This reverts commit a7854487cd7128a30a7f4f5259de9f67d5efb95f.
The above patch would cause lockups of RAID4/5 arrays if a drive is
going faulty during reconstruction.
Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com>
---
drivers/md/raid5.c | 19 +++----------------
1 file changed, 3 insertions(+), 16 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index aa76865..bc3a085 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -3159,25 +3159,12 @@ static void handle_stripe_dirtying(struct r5conf *conf,
int disks)
{
int rmw = 0, rcw = 0, i;
- sector_t recovery_cp = conf->mddev->recovery_cp;
-
- /* RAID6 requires 'rcw' in current implementation.
- * Otherwise, check whether resync is now happening or should start.
- * If yes, then the array is dirty (after unclean shutdown or
- * initial creation), so parity in some stripes might be inconsistent.
- * In this case, we need to always do reconstruct-write, to ensure
- * that in case of drive failure or read-error correction, we
- * generate correct data from the parity.
- */
- if (conf->max_degraded == 2 ||
- (recovery_cp < MaxSector && sh->sector >= recovery_cp)) {
- /* Calculate the real rcw later - for now make it
+ if (conf->max_degraded == 2) {
+ /* RAID6 requires 'rcw' in current implementation
+ * Calculate the real rcw later - for now fake it
* look like rcw is cheaper
*/
rcw = 1; rmw = 2;
- pr_debug("force RCW max_degraded=%u, recovery_cp=%llu sh->sector=%llu\n",
- conf->max_degraded, (unsigned long long)recovery_cp,
- (unsigned long long)sh->sector);
} else for (i = disks; i--; ) {
/* would I have to read this buffer for read_modify_write */
struct r5dev *dev = &sh->dev[i];
--
1.8.3.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox