* Re: md failing mechanism
From: Phil Turmel @ 2016-01-23 0:44 UTC (permalink / raw)
To: James J, Dark Penguin, linux-raid
In-Reply-To: <56A2BDF7.7020101@shiftmail.org>
On 01/22/2016 06:40 PM, James J wrote:
> On 22/01/2016 22:44, Dark Penguin wrote:
>>
>> As I understand, one way around this problem is to change the kernel
>> timeout to exceed the drive timeout by changing
>> /sys/block/sd?/device/timeout to something larger than the default 30,
>> but I'd have to do that after every reboot, is all that correct?
>>
>
> No, this part needs further investigation and comments from the gurus.
Yes, DP had that correct.
> With a SCSI timeout 30 secs, which is the setting you had at the time of
> the incident AFAIU, what should have happened was that the drive should
> have been kicked out at the 30th second, this is BEFORE it had a chance
> to return a read failure because your desktop drive takes more than
> 30secs to return a read failure. This was what you indeed expected but
> it is not what has happened.
His problem description doesn't perfectly match timeout mismatch. He
probably had a real problem that was exacerbated by his now-discovered
timeout problem. He no longer has the dmesg so further speculation is
moot. If it happens again, we can look closer.
> The recommentation of raising the timeout to 120+ is for the opposite
> purpose of what you want. It is for the case the sysadmin accepts to
> wait a long time because he wants to prevent the kicking of the drive at
> the first read-error (normally drives are kicked for a write error).
> This might be wanted in order to a) defer the replacement of the drive,
> either to perform the replacement at a more opportune time and/or in a
> better manner such as a no-degrade replace operation, or b) because he
> does not want to replace the drive at all: maybe he believes that the
> error might be spurious and will not happen again and the drive is still
> of acceptable fitness for the purpose, e.g. in a low-cost file server.
No. If you have a drive that doesn't support scterc or has it turned
off, you *must* set a timeout longer than the drive's native timeout or
you will have great problems. I suggest you read the references to the
archives I posted.
Keep in mind that in a properly working array UREs are *fixed* when
discovered by overwriting them. This is vital to array robustness, as
many UREs are transient (don't need relocation at all).
Phil
^ permalink raw reply
* Re: md failing mechanism
From: Phil Turmel @ 2016-01-23 0:34 UTC (permalink / raw)
To: Dark Penguin, Edward Kuns, linux-raid
In-Reply-To: <56A2B639.7000902@yandex.ru>
On 01/22/2016 06:07 PM, Dark Penguin wrote:
>> Depending on your distribution, you may have been scrubbing all along.
>
>> Notice that if you have problems with timeouts, then this scrubbing
>> can break your array by causing you to hit a bad sector and fail as
>> Phil and others have described in several of his referenced EMails.
>
> I remember disabling scrubbing myself. My reasons were not very...
> bright, but now it turned out to be a it's a good thing, because with
> TL;DR disabled by default, it could lead to that kind of bad things
> happening, yes.
Nooooo!
Disabling scrubbing with "dodgy" drives turns an annoyance (drives
kicked out by scrubs) into a catastrophe later (unrecoverable array).
The archives of this list are full of such incidents.
The only people who should disable scrubbing are enterprise data centers
where the disks are so busy they hit every sector every so often anyways.
> I remember having one drive kicked out of an array in my
> home storage, and since then, I've learned to use write-intent bitmaps
> to re-add them more easily. But I'm a BAARF person, so I only have
> mirrors; I wonder what happens if the only drive in a degraded mirror
> fails?..
Or any drive in a raid5 hits an error while replacing a failed disk.
BOOM.
Phil
^ permalink raw reply
* Re: md failing mechanism
From: Dark Penguin @ 2016-01-23 0:09 UTC (permalink / raw)
To: Wols Lists, Edward Kuns, linux-raid
In-Reply-To: <56A2BDAA.9070900@youngman.org.uk>
>> I was afraid to learn that my latest pack of drives are modern enough to
>> be castrated already, but it turned out they are not. :) My anime is
>> safe with the last 5900RPM Hitachi drives available, and they do support
>> TL;DR. I wish Hitachi made more 5900RPM drives for such purposes...
>
> I have a gut feel that drives of 1TB or less are "old technology" and
> even new today are okay. It's the multi-TB drives that have been
> castrated...
>
> Note I said "gut feel" - don't trust what I think ... :-)
I'm not sure exactly when did the "castration" started, but I have few
Seagate LP's 1,5 and 2 Tb, over 5 years old: they support it. But I
found out that they suck in terms of reliability, so I switched to
Hitachi. There was this 5900RPM model about four years ago, I think - I
believe it was almost the only low-RPM one Hitachi ever did; they
support it, too. And I have a 500Gb 2,5" Hitachi Travelstar bought a
year or two later, and even they support it! So, I guess I was right;
apparently, Hitachi is not into the castration business... Has anybody
seen any Hitachi without TL;DR ?..
--
darkpenguin
^ permalink raw reply
* Re: md failing mechanism
From: Dark Penguin @ 2016-01-23 0:09 UTC (permalink / raw)
To: Wols Lists, Phil Turmel, linux-raid
In-Reply-To: <56A2BCA1.6070005@youngman.org.uk>
>> Since that was written, 'A' would now include almost-enterprise drives
>> with RAID ratings like the Western Digital Red family. And the
>> recommended timeout for 'C' has drifted upward to 180.
>
> Almost all posts mention WD Reds. It's NOT a recommendation, but seeing
> as I've tended to buy Seagate (note my Barracudas) I looked for the
> Seagate equivalent. They're called NAS drives, and they cost roughly the
> same ... just pointing this out in case people like Seagate or dislike
> WD ... :-)
I would prefer a Seagate over a WD, but with my experience lately, I
don't want any more Seagates, either... I switched to Hitachi, but they
don't make low-speed drives anymore, and they are expensive... I wonder
of those Seagate NAS drives are actually anywhere near "reliable"?..
--
darkpenguin
^ permalink raw reply
* Re: md failing mechanism
From: James J @ 2016-01-22 23:40 UTC (permalink / raw)
To: Dark Penguin, linux-raid
In-Reply-To: <56A2A2C3.9000801@yandex.ru>
On 22/01/2016 22:44, Dark Penguin wrote:
>
> As I understand, one way around this problem is to change the kernel
> timeout to exceed the drive timeout by changing
> /sys/block/sd?/device/timeout to something larger than the default 30,
> but I'd have to do that after every reboot, is all that correct?
>
No, this part needs further investigation and comments from the gurus.
With a SCSI timeout 30 secs, which is the setting you had at the time of
the incident AFAIU, what should have happened was that the drive should
have been kicked out at the 30th second, this is BEFORE it had a chance
to return a read failure because your desktop drive takes more than
30secs to return a read failure. This was what you indeed expected but
it is not what has happened.
The recommentation of raising the timeout to 120+ is for the opposite
purpose of what you want. It is for the case the sysadmin accepts to
wait a long time because he wants to prevent the kicking of the drive at
the first read-error (normally drives are kicked for a write error).
This might be wanted in order to a) defer the replacement of the drive,
either to perform the replacement at a more opportune time and/or in a
better manner such as a no-degrade replace operation, or b) because he
does not want to replace the drive at all: maybe he believes that the
error might be spurious and will not happen again and the drive is still
of acceptable fitness for the purpose, e.g. in a low-cost file server.
So what happened is still wrong AFAIK, in the sense of a kernel bug.
^ permalink raw reply
* Re: md failing mechanism
From: Wols Lists @ 2016-01-22 23:39 UTC (permalink / raw)
To: Dark Penguin, Edward Kuns, linux-raid
In-Reply-To: <56A2B639.7000902@yandex.ru>
On 22/01/16 23:07, Dark Penguin wrote:
> I was afraid to learn that my latest pack of drives are modern enough to
> be castrated already, but it turned out they are not. :) My anime is
> safe with the last 5900RPM Hitachi drives available, and they do support
> TL;DR. I wish Hitachi made more 5900RPM drives for such purposes...
I have a gut feel that drives of 1TB or less are "old technology" and
even new today are okay. It's the multi-TB drives that have been
castrated...
Note I said "gut feel" - don't trust what I think ... :-)
Cheers,
Wol
^ permalink raw reply
* Re: md failing mechanism
From: Wols Lists @ 2016-01-22 23:34 UTC (permalink / raw)
To: Phil Turmel, Dark Penguin, linux-raid
In-Reply-To: <56A2AAB1.6070305@turmel.org>
On 22/01/16 22:18, Phil Turmel wrote:
> Since that was written, 'A' would now include almost-enterprise drives
> with RAID ratings like the Western Digital Red family. And the
> recommended timeout for 'C' has drifted upward to 180.
Almost all posts mention WD Reds. It's NOT a recommendation, but seeing
as I've tended to buy Seagate (note my Barracudas) I looked for the
Seagate equivalent. They're called NAS drives, and they cost roughly the
same ... just pointing this out in case people like Seagate or dislike
WD ... :-)
Cheers,
Wol
^ permalink raw reply
* Re: md failing mechanism
From: Edward Kuns @ 2016-01-22 23:23 UTC (permalink / raw)
To: Dark Penguin; +Cc: Phil Turmel, Linux-RAID
In-Reply-To: <56A2B223.7060804@yandex.ru>
On Fri, Jan 22, 2016 at 4:50 PM, Dark Penguin <darkpenguin@yandex.ru> wrote:
>> the recommended timeout for 'C' has drifted upward to 180.
>
> Yes, I saw this; but, is it really not possible to examine the default
> timeout in a certain desktop drive, rather than follow rough estimates like
> "about two of three minutes should be enough"?
When I asked something similar in
https://www.marc.info/?l=linux-raid&m=144666581206186&w=2 the answers
I got
https://www.marc.info/?l=linux-raid&m=144666614306348&w=2
https://www.marc.info/?l=linux-raid&m=144666776906740&w=2
indicated that you just need the kernel timeout to be longer than the
disk timeout. *Anything* longer is good. It just has to be longer.
You want the *disk* to return a complaint. You don't want the driver
to time out before the disk has had the opportunity to complain. The
point of setting SCTERC to 7 seconds is to force the *drive* (not
kernel) to give up quickly and complain. That'll force a rebuild,
rewriting the sector in question and remapping the bad sector on the
drive with the failure. (If I've misunderstood this hopefully someone
much more expert will correct me!) If you can't configure the drive
to give up quickly on a read error, then your remaining alternative is
to tell the kernel to wait (effectively) forever (180 sec) to allow
the drive enough time to give up on its own.
Regarding disabling scrubbing, I'd say that even with disabled SCTERC
(TL;DR haha) and bad default timeouts, it's still better for the array
to fail right away when you only have a single bad sector on a single
drive. If you let it wait, you run the risk of developing a bad
sector on another drive, so that when you hit the first bad sector
through usage and the drive fails and you trigger a rebuild, you hit
the bad sector on the other drive and *bam* you're toast.
Eddie
^ permalink raw reply
* Re: Wrong device name after hot-swap
From: Paul van der Vlis @ 2016-01-22 23:15 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <56A2ACB4.6060808@turmel.org>
Op 22-01-16 om 23:27 schreef Phil Turmel:
> On 01/22/2016 04:55 PM, Paul van der Vlis wrote:
>> Hello,
>>
>> I want to put bigger disks into my server. What I want to do is replace
>> the first disk, rebuild the raid, replace the second disk, rebuild the
>> raid. The machine has two disks, sda and sdb.
>>
>> But, when I replace a disk, it gets a new device name. E.g. /dev/sdb
>> becomes /dev/sdc. After a reboot it's good again, but I prefer not to
>> reboot this machine!
>>
>> Is there a way to get the correct device name?
>
> No. Device names are assigned in the order they are encountered after
> boot, and that order is not guaranteed by the kernel. You should never
> depend on those names.
>
> When a device name is fully disconnected, modern kernels will recycle
> the name at the next opportunity.
Correct, I see that with USB-sticks.
> You must be using a hotplug-enabled
> driver. For most motherboards, turning on "AHCI" mode in the BIOS on
> those sata ports is all you need.
I have that. I am using an Intel chipset:
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset
Family SATA AHCI Controller (rev 05)
>> When not:
>> Is it maybe an idea too add the wrong device name to the md-device?
>> mdadm /dev/md0 -a /dev/sdc1
>> Then replace /dev/sda what becomes /dev/sdd:
>> mdadm /dev/md0 -a /dev/sdd1
>> Then restore grub on both disks (with "--recheck" ??)
>> But what will happen after a reboot later? Will the md-device be
>> restored with the old names?
>
> MD stores signatures in the devices it uses that identify them for later
> assembly. It does not depend on the device name, though it is recorded
> in the superblock as a "last connected as" kind of indicator.
So I can do it as described above?
> In general, you should not rely on device names in your system
> configuration. UUIDs and filesystem labels were implemented
> specifically to avoid this problem.
So far I know I cannot configure anywhere a disk UUID or a filesytem
label to a MD-device.
It feels wrong when I add /dev/sdc1 to the raid, when the name is
normally /dev/sdb1. But maybe it's no problem, because the device name
is not impartant while booting from an MD-device.
With regards,
Paul van der VLis.
--
Paul van der Vlis Linux systeembeheer Groningen
https://www.vandervlis.nl/
^ permalink raw reply
* Re: md failing mechanism
From: Dark Penguin @ 2016-01-22 23:07 UTC (permalink / raw)
To: Edward Kuns, linux-raid
In-Reply-To: <CACsGCyTUBRrQgt0GiVp1R6pv-4gO44Ro7pgm4iCq+1KwWDoPsg@mail.gmail.com>
> Depending on your distribution, you may have been scrubbing all along.
> Notice that if you have problems with timeouts, then this scrubbing
> can break your array by causing you to hit a bad sector and fail as
> Phil and others have described in several of his referenced EMails.
I remember disabling scrubbing myself. My reasons were not very...
bright, but now it turned out to be a it's a good thing, because with
TL;DR disabled by default, it could lead to that kind of bad things
happening, yes. I remember having one drive kicked out of an array in my
home storage, and since then, I've learned to use write-intent bitmaps
to re-add them more easily. But I'm a BAARF person, so I only have
mirrors; I wonder what happens if the only drive in a degraded mirror
fails?..
> I took the script from this Email:
>
> https://www.marc.info/?l=linux-raid&m=144661276420400&w=2
>
> and dropped that code in my /etc/rc.d/rc.local after verifying that my
> Linux distribution still ran that script on every startup. YMMV.
> That solved my problem. Good luck.
I was afraid to learn that my latest pack of drives are modern enough to
be castrated already, but it turned out they are not. :) My anime is
safe with the last 5900RPM Hitachi drives available, and they do support
TL;DR. I wish Hitachi made more 5900RPM drives for such purposes...
--
darkpenguin
^ permalink raw reply
* Re: md failing mechanism
From: Dark Penguin @ 2016-01-22 22:50 UTC (permalink / raw)
To: Phil Turmel, linux-raid
In-Reply-To: <56A2AAB1.6070305@turmel.org>
>> C) Suffer with desktop drives without SCTERC support. They cannot be
>> set to appropriate error timeouts. Udev or boot script assistance is
>> needed to set a 120 second driver timeout in sysfs. They do *not* work
>> properly with MD out of the box.
> the recommended timeout for 'C' has drifted upward to 180.
Yes, I saw this; but, is it really not possible to examine the default
timeout in a certain desktop drive, rather than follow rough estimates
like "about two of three minutes should be enough"?.. I wanted to make
sure it is indeed not possible, because that is hard to believe. Or do
they not have a specified timeout at all?..
> Since that was written, 'A' would now include almost-enterprise drives
> with RAID ratings like the Western Digital Red family.
Yes, I understand that; always make sure they support it.
>> Still, I don't think it has anything to do with what has happened to my
>> "small file server"...
>
> That's why I asked for the dmesg. It could have been a bug. No crisis
> if it's lost, so long as you've accepted one of A through D above.
I've moved all the data to another server, disassembled this one, and
reused the surviving hard drive, so I'm safe, but sadly, no logs. The
important thing is, I've confirmed that this is not the expected
behaviour - I was kind of ready to hear that "that's how it is with
softraids, faulty drives hang your entire system like they hang Windows".
I've checked all my hard drives in all my RAIDs; all of them support
"TL;DR" technology. They were all made before the "crippling" tendencies
took over, and they are mostly Hitachi, so I'm lucky.
--
darkpenguin
^ permalink raw reply
* Re: md failing mechanism
From: Edward Kuns @ 2016-01-22 22:37 UTC (permalink / raw)
To: Dark Penguin; +Cc: Linux-RAID
In-Reply-To: <56A2A2C3.9000801@yandex.ru>
On Fri, Jan 22, 2016 at 3:44 PM, Dark Penguin <darkpenguin@yandex.ru> wrote:
> And also, now I understand why I probably "should have been scrubbing". =/
Depending on your distribution, you may have been scrubbing all along.
When I looked into this, I discovered that mdadm as bundled in Fedora
(at least as of 21) already scrubs weekly:
$ rpm -ql mdadm | grep /etc
/etc/cron.d/raid-check
/etc/libreport/events.d/mdadm_event.conf
/etc/sysconfig/raid-check
$ cat /etc/cron.d/raid-check
# Run system wide raid-check once a week on Sunday at 1am by default
0 1 * * Sun root /usr/sbin/raid-check
Sweet! I was pleased to discovered this when I realized I had been
lax. The /usr/sbin/raid-check script does scrubbing, as configured by
/etc/sysconfig/raid-check. I looked in /var/log/messages and indeed
saw evidence of successful weekly scrubbing.
Notice that if you have problems with timeouts, then this scrubbing
can break your array by causing you to hit a bad sector and fail as
Phil and others have described in several of his referenced EMails.
But better to fail early while only one drive is bad than to discover
this after more than one drive has problems and your data is
irrecoverable. I had a mirror where one drive kept falling out. I
now understand why. (Weekly scrubbing + dodgy drives + no attempt to
address the timeouts == occasional unnecessary failure.)
> As I understand, one way around this problem is to change the kernel timeout
> to exceed the drive timeout by changing /sys/block/sd?/device/timeout to
> something larger than the default 30, but I'd have to do that after every
> reboot, is all that correct?
I took the script from this Email:
https://www.marc.info/?l=linux-raid&m=144661276420400&w=2
and dropped that code in my /etc/rc.d/rc.local after verifying that my
Linux distribution still ran that script on every startup. YMMV.
That solved my problem. Good luck.
Eddie
^ permalink raw reply
* Re: Wrong device name after hot-swap
From: Phil Turmel @ 2016-01-22 22:27 UTC (permalink / raw)
To: Paul van der Vlis, linux-raid
In-Reply-To: <n7u8gr$1bv$1@ger.gmane.org>
On 01/22/2016 04:55 PM, Paul van der Vlis wrote:
> Hello,
>
> I want to put bigger disks into my server. What I want to do is replace
> the first disk, rebuild the raid, replace the second disk, rebuild the
> raid. The machine has two disks, sda and sdb.
>
> But, when I replace a disk, it gets a new device name. E.g. /dev/sdb
> becomes /dev/sdc. After a reboot it's good again, but I prefer not to
> reboot this machine!
>
> Is there a way to get the correct device name?
No. Device names are assigned in the order they are encountered after
boot, and that order is not guaranteed by the kernel. You should never
depend on those names.
When a device name is fully disconnected, modern kernels will recycle
the name at the next opportunity. You must be using a hotplug-enabled
driver. For most motherboards, turning on "AHCI" mode in the BIOS on
those sata ports is all you need.
> When not:
> Is it maybe an idea too add the wrong device name to the md-device?
> mdadm /dev/md0 -a /dev/sdc1
> Then replace /dev/sda what becomes /dev/sdd:
> mdadm /dev/md0 -a /dev/sdd1
> Then restore grub on both disks (with "--recheck" ??)
> But what will happen after a reboot later? Will the md-device be
> restored with the old names?
MD stores signatures in the devices it uses that identify them for later
assembly. It does not depend on the device name, though it is recorded
in the superblock as a "last connected as" kind of indicator.
In general, you should not rely on device names in your system
configuration. UUIDs and filesystem labels were implemented
specifically to avoid this problem.
Phil
^ permalink raw reply
* Re: md failing mechanism
From: Phil Turmel @ 2016-01-22 22:18 UTC (permalink / raw)
To: Dark Penguin, linux-raid
In-Reply-To: <56A2A2C3.9000801@yandex.ru>
On 01/22/2016 04:44 PM, Dark Penguin wrote:
> Oh! Thank you! I really wanted to see a reliable "what's supposed to
> happen" sequence!
You're welcome.
> As for my case, those were indeed, um, "cheap desktop drives" - to be
> precise, some 80-Gb IDE drives in a Pentium-4 machine; "it works well
> for a small file server", I thought, oblivious to the finer details
> about the process of failure handling... But, I also have "big" file
> servers, so that timeout mismatch issue is something worth paying
> attention!
>
> And also, now I understand why I probably "should have been scrubbing".
> =/ Do I understand correctly that "scrubbing" means those "monthly
> redundancy checks" that mdadm suggests? And I suppose what it does is
> just the same - read every sector and attempt to write it back upon
> failure, otherwise kicking the device?
A "check" scrub reads every sector every member device's data area. If
any fail, the normal reconstruct and rewrite will fix it. It also looks
for successfull reads where the data is inconsistent between mirrors or
between data blocks and parity blocks. Those are counted for you to review.
A "repair" scrub reads forcibly ensures consistent redundancy by copying
mirror one to the others, and recomputing parity from data. It will
also reconstruct if needed.
The "check" mode is your recommended regular scrub. I do mine weekly,
but monthly is probably fine. "Repair" is needed if "check" reports any
mismatches.
> ..... is all that correct?
From one of your reading assignments: (
http://marc.info/?l=linux-raid&m=135811522817345&w=1 )
> Options are:
>
> A) Buy Enterprise drives. They have appropriate error timeouts and work
> properly with MD right out of the box.
>
> B) Buy Desktop drives with SCTERC support. They have inappropriate
> default timeouts, but can be set to an appropriate value. Udev or boot
> script assistance is needed to call smartctl to set it. They do *not*
> work properly with MD out of the box.
>
> C) Suffer with desktop drives without SCTERC support. They cannot be
> set to appropriate error timeouts. Udev or boot script assistance is
> needed to set a 120 second driver timeout in sysfs. They do *not* work
> properly with MD out of the box.
>
> D) Lose your data during spare rebuild after your first URE. (Odds in
> proportion to array size.)
>
> One last point bears repeating: MD is *not* a backup system, although
> some people leverage it's features for rotating off-site backup disks.
> Raid arrays are all about *uptime*. They will not save you from
> accidental deletion or other operator errors. They will not save you if
> your office burns down. You need a separate backup system for critical
> files.
Since that was written, 'A' would now include almost-enterprise drives
with RAID ratings like the Western Digital Red family. And the
recommended timeout for 'C' has drifted upward to 180.
[trim /]
> Still, I don't think it has anything to do with what has happened to my
> "small file server"...
That's why I asked for the dmesg. It could have been a bug. No crisis
if it's lost, so long as you've accepted one of A through D above.
Phil
ps. convention on kernel.org is reply-to-all and no top-posting.
^ permalink raw reply
* Wrong device name after hot-swap
From: Paul van der Vlis @ 2016-01-22 21:55 UTC (permalink / raw)
To: linux-raid
Hello,
I want to put bigger disks into my server. What I want to do is replace
the first disk, rebuild the raid, replace the second disk, rebuild the
raid. The machine has two disks, sda and sdb.
But, when I replace a disk, it gets a new device name. E.g. /dev/sdb
becomes /dev/sdc. After a reboot it's good again, but I prefer not to
reboot this machine!
Is there a way to get the correct device name?
When not:
Is it maybe an idea too add the wrong device name to the md-device?
mdadm /dev/md0 -a /dev/sdc1
Then replace /dev/sda what becomes /dev/sdd:
mdadm /dev/md0 -a /dev/sdd1
Then restore grub on both disks (with "--recheck" ??)
But what will happen after a reboot later? Will the md-device be
restored with the old names?
With regards,
Paul van der Vlis.
--
Paul van der Vlis Linux systeembeheer Groningen
https://www.vandervlis.nl/
^ permalink raw reply
* Re: md failing mechanism
From: Dark Penguin @ 2016-01-22 21:44 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <56A28309.9080806@turmel.org>
Oh! Thank you! I really wanted to see a reliable "what's supposed to
happen" sequence!
As for my case, those were indeed, um, "cheap desktop drives" - to be
precise, some 80-Gb IDE drives in a Pentium-4 machine; "it works well
for a small file server", I thought, oblivious to the finer details
about the process of failure handling... But, I also have "big" file
servers, so that timeout mismatch issue is something worth paying attention!
And also, now I understand why I probably "should have been scrubbing".
=/ Do I understand correctly that "scrubbing" means those "monthly
redundancy checks" that mdadm suggests? And I suppose what it does is
just the same - read every sector and attempt to write it back upon
failure, otherwise kicking the device?
So, I understand a common problem now: the read timeout on the "desktop"
drives is too long, which makes sense for the desktops, but not for
RAIDs, because the "write back attempt" fails and leads to "BOOM" and
kick. Enterprise-grade drives, however, offer an option to change their
timeout, which is called "TL;DR technology" (yes, that's how I'm going
to call it! Because I can't remember the acronym no matter how may times
I read it, and the meaning kinda fits!). And what about drives that do
not support it?.. Do they even have some kid of huge timeout or
something?.. Yesterday I've been checking one drive for bad blocks
(badblocks read-only test), and it took no more than two seconds per
block to confirm its... badness!
As I understand, one way around this problem is to change the kernel
timeout to exceed the drive timeout by changing
/sys/block/sd?/device/timeout to something larger than the default 30,
but I'd have to do that after every reboot, is all that correct?
Still, I don't think it has anything to do with what has happened to my
"small file server"... It was the opposite; for some reason, it was not
kicked from the array. But, it happened a while ago, and I've destroyed
the array afterwards, so I can't get any more data about that incident.
But, I've got what I wanted: I now I know what is supposed to happen
when a drive in a RAID fails, and it's not what happened that time. And
I know I should set up proper TL;DR timeouts and scrubbing...
--
darkpenguin
^ permalink raw reply
* Re: md failing mechanism
From: Wols Lists @ 2016-01-22 20:00 UTC (permalink / raw)
To: Dark Penguin, linux-raid
In-Reply-To: <56A28309.9080806@turmel.org>
On 22/01/16 19:29, Phil Turmel wrote:
> What you've described does sound like a bug, maybe. It also sounds
> similar to traditional timeout mismatch caused by cheap desktop drives
> used in a raid array.
man smartctl
smartctl -i /dev/sd-dodgy-drive
If I run that on my drives I get
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.1.12-gentoo] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-1CH166
Serial Number: W1F50K0F
LU WWN Device Id: 5 000c50 0737a98a4
Firmware Version: CC27
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Fri Jan 22 19:56:47 2016 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Disabled
NOTE THAT SMART SUPPORT IS DISABLED.
Yes I know I'm running with dodgy drives and a dodgy config, but smart
support should be enabled and things like ERC should be turned on. If
they're not ...
Cheers,
Wol
^ permalink raw reply
* Re: md failing mechanism
From: Phil Turmel @ 2016-01-22 19:29 UTC (permalink / raw)
To: Dark Penguin, linux-raid
In-Reply-To: <56A26E11.2090703@yandex.ru>
Hi,
On 01/22/2016 12:59 PM, Dark Penguin wrote:
> Greetings,
>
> Recently, I've had my first drive failure in a software RAID1 on a file
> server. And I was really surprised about exactly what happened; I always
> thought that when md can't process a read request from one of the
> drives, it is supposed to mark that drive as faulty and read from
> another drive; but, for some reason, it was deliberately trying to read
> from a faulty drive no matter what, which apparently caused Samba to
> wait until it's finished, and so the whole server was rendered
> inaccessible (I mean, the whole Samba).
What you've described does sound like a bug, maybe. It also sounds
similar to traditional timeout mismatch caused by cheap desktop drives
used in a raid array.
In a properly functioning array, the normal sequence of events for a
simple failing sector is:
1) read from sector X fails and is reported by the drive to the kernel
2) kernel tells MD "read failed"
3) MD reads from different mirror or from peers & parity to reconstruct
the failed sector
4a) MD supplies reconstructed sector to upper layer/user.
4b) MD writes reconstructed sector back to failed location to fix it or
relocate it. If this write succeeds (either case), the device stays in
the array.
The above sequence of events is disturbed when a drive takes too long in
step 1.
It would be good to see your dmesg of this event to see what failure
mode is present.
Meanwhile, some reading material for you:
http://marc.info/?l=linux-raid&m=139050322510249&w=2
http://marc.info/?l=linux-raid&m=135863964624202&w=2
http://marc.info/?l=linux-raid&m=135811522817345&w=1
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=132477199207506
http://marc.info/?l=linux-raid&m=133665797115876&w=2
http://marc.info/?l=linux-raid&m=142487508806844&w=3
http://marc.info/?l=linux-raid&m=144535576302583&w=2
^ permalink raw reply
* Re: RAID 6 "Failed to restore critical section for reshape, sorry." - recovery advice?
From: George Rapp @ 2016-01-22 19:24 UTC (permalink / raw)
To: Linux-RAID
In-Reply-To: <87r3hyxf69.fsf@notabene.neil.brown.name>
On Sun, Jan 3, 2016 at 9:16 PM, NeilBrown <neilb@suse.com> wrote:
> On Sun, Dec 27 2015, George Rapp wrote:
>
> If you apply the following patch to mdadm (
> git clone git://neil.brown.name/mdadm
> apply patch
> make
> ) and then try to assemble with --update=revert-reshape, it should
> assemble as a 5-device array with no reshape happening.
>
> I probably want more safety checks before this goes upstream, but it
> is safe enough for you.
>
> [patch snipped]
>
I've replied privately to Neil, but for list archive purposes, his
patch solved my problem.
After applying his patch, I used the custom-build version of mdadm to
assemble my array:
# UUID=$(mdadm -E /dev/sdd4 | perl -ne '/Array UUID : (\S+)/ and print $1')
# DEVICES=$(cat /proc/partitions | parallel --tagstring {5} --colsep '
+' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t' echo
/dev/{1})
# ./mdadm --assemble --verbose --update=revert-reshape /dev/md4 $DEVICES
mdadm: looking for devices for /dev/md4
mdadm: /dev/sdd4 is identified as a member of /dev/md4, slot 0.
mdadm: /dev/sdc4 is identified as a member of /dev/md4, slot 1.
mdadm: /dev/sdg4 is identified as a member of /dev/md4, slot 2.
mdadm: /dev/sdh1 is identified as a member of /dev/md4, slot 3.
mdadm: /dev/sdi1 is identified as a member of /dev/md4, slot 5.
mdadm: /dev/sdj1 is identified as a member of /dev/md4, slot 4.
mdadm: device 10 in /dev/md4 has wrong state in superblock, but
/dev/sdi1 seems ok
mdadm: added /dev/sdc4 to /dev/md4 as 1
mdadm: added /dev/sdg4 to /dev/md4 as 2
mdadm: added /dev/sdh1 to /dev/md4 as 3
mdadm: added /dev/sdj1 to /dev/md4 as 4
mdadm: added /dev/sdi1 to /dev/md4 as 5
mdadm: added /dev/sdd4 to /dev/md4 as 0
mdadm: /dev/md4 has been started with 5 drives and 1 spare.
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md4 : active raid6 sdd4[0] sdi1[8](S) sdj1[7] sdh1[6] sdg4[5] sdc4[1]
5756723712 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]
bitmap: 0/15 pages [0KB], 65536KB chunk
For future reference, I would highly recommend executing
# setenforce 0
or, if you're really paranoid, disabling SELinux completely
- edit /etc/sysconfig/selinux and add or modify this line:
SELINUX=disabled
- # touch /.relabel
- # systemctl reboot
(and go get a cup of coffee while your filesystem gets relabeled)
before making any metadata changes on a RAID 6 array.
Thanks again, Neil and members of the Linux-RAID community!
--
George Rapp (Pataskala, OH) Home: george.rapp -- at -- gmail.com
LinkedIn profile: https://www.linkedin.com/in/georgerapp
Phone: +1 740 936 RAPP (740 936 7277)
^ permalink raw reply
* Re: broken raid level 5 array caused by user error
From: Phil Turmel @ 2016-01-22 19:13 UTC (permalink / raw)
To: Mathias Mueller; +Cc: Linux raid, linux-raid-owner
In-Reply-To: <9ce1b753a3839866f6ea309465e1fec2@pingofdeath.de>
On 01/22/2016 12:39 PM, Mathias Mueller wrote:
> Hi Phil,
>
> you are the man! The first one is working!
Ah, that's good.
> So please tell me, where does the beer delivery goes to? :)
Sadly, it's hard to ship beer here in the US without government involvement.
> Thank you so much, especially for so much patience with the dumbest user
> on earth.
You're welcome. But don't put yourself down -- it was your idea to
search for parts of a known file, after all.
> Mathias
>
> PS: I'm serious with the beer or whatever you prefer
A modest tip via paypal would be fine, if you insist. Or something from
my public wishlist:
http://smile.amazon.com/registry/wishlist/2EE91JWNIYSC3
Phil
^ permalink raw reply
* md failing mechanism
From: Dark Penguin @ 2016-01-22 17:59 UTC (permalink / raw)
To: linux-raid
Greetings,
Recently, I've had my first drive failure in a software RAID1 on a file
server. And I was really surprised about exactly what happened; I always
thought that when md can't process a read request from one of the
drives, it is supposed to mark that drive as faulty and read from
another drive; but, for some reason, it was deliberately trying to read
from a faulty drive no matter what, which apparently caused Samba to
wait until it's finished, and so the whole server was rendered
inaccessible (I mean, the whole Samba).
What I expected:
- A user tries to read a file via Samba.
- Samba issues a read request to md.
- md tries to read the file from one of the drives... the drive is
struggling to read a bad sector...
- md thinks: okay, this is taking too long, production is not waiting;
I'll just read from another drive instead.
- It reads from another drive successfully, and users continue their work.
- Finally, the "bad" drive gives up on trying to read the bad sector and
returns an error. md marks the drive as faulty and sends an email
telling me to replace the drive as soon as possible.
What happened instead:
- A user tries to read a file via Samba.
- Samba issues a read request to md.
- md tries to read the file from one of the drives... the drive is
struggling to read a bad sector... Samba is waiting for md, md is
waiting for the drive, and the drive is trying again and again to read
this blasted sector like its life depends on it, while users see that
the network folder doesn't respond anymore at all.
This goes on forever, until users call me, I come to investigate, see
Samba down, see a lot of errors in dmesg, and then I manually mark this
drive as faulty.
Now, that happened a while ago; I did not have the most recent kernel on
that server (I think it was 3.2 from Debian Wheezy or something a little
newer from the backports), but I can't try it again with a new server,
because I can't make a functional RAID1, write data there, and then
destroy some sectors and see what happens. I just want to ask, is that
really how it works?.. Was that supposed to happen?.. I thought the main
point of a RAID1 is to avoid any downtime, especially in such cases!..
Or is it maybe a known issue fixed in the more recent versions, so I
should just update my kernels and expect different behaviour next time?..
--
darkpenguin
^ permalink raw reply
* Re: broken raid level 5 array caused by user error
From: Mathias Mueller @ 2016-01-22 17:39 UTC (permalink / raw)
To: Phil Turmel; +Cc: Linux raid, linux-raid-owner
In-Reply-To: <56A263F4.8020307@turmel.org>
Hi Phil,
you are the man! The first one is working!
So please tell me, where does the beer delivery goes to? :)
Thank you so much, especially for so much patience with the dumbest user
on earth.
Mathias
PS: I'm serious with the beer or whatever you prefer
^ permalink raw reply
* Re: broken raid level 5 array caused by user error
From: Phil Turmel @ 2016-01-22 17:16 UTC (permalink / raw)
To: Mathias Mueller; +Cc: Linux raid, linux-raid-owner
In-Reply-To: <34ad7f74bc9fdc37aab022054d8d856e@pingofdeath.de>
Hi Mathias,
On 01/22/2016 04:30 AM, Mathias Mueller wrote:
> Hi Phil,
>
> it's done:
>
> 658 Unique hashes in picture.jpg
Looks good. Rearranging a bit:
> /dev/sdf1 @ 005400caa000:005400cfffff ~= 00000000:00055fff
> /dev/sde1 @ 005400b90000:005400c0ffff ~= 00056000:000d5fff
> /dev/sdd1 @ 005400c80000:005400cfffff ~= 000d6000:00155fff
> /dev/sdc1 @ 005400d00000:005400d7ffff ~= 00156000:001d5fff
> /dev/sdf1 @ 005400d00000:005400d7ffff ~= 001d6000:00255fff
> /dev/sde1 @ 005400c10000:005400c4bfff ~= 00256000:00291fff
> /dev/sdc1 @ 0087b4f29000:0087b4f7ffff ~= 00000000:00056fff
> /dev/sdf1 @ 0087b4f00000:0087b4f7ffff ~= 00057000:000d6fff
> /dev/sde1 @ 0087b4e10000:0087b4e8ffff ~= 000d7000:00156fff
> /dev/sdd1 @ 0087b4f80000:0087b4ffffff ~= 00157000:001d6fff
> /dev/sdc1 @ 0087b4f80000:0087b4ffffff ~= 001d7000:00256fff
> /dev/sdf1 @ 0087b4f80000:0087b4fbafff ~= 00257000:00291fff
> /dev/sdf1 @ 00cb2d956000:00cb2d97ffff ~= 00000000:00029fff
> /dev/sde1 @ 00cb2d810000:00cb2d88ffff ~= 0002a000:000a9fff
> /dev/sdd1 @ 00cb2d980000:00cb2d9fffff ~= 000aa000:00129fff
> /dev/sdc1 @ 00cb2d980000:00cb2d9fffff ~= 0012a000:001a9fff
> /dev/sdf1 @ 00cb2d980000:00cb2d9fffff ~= 001aa000:00229fff
> /dev/sde1 @ 00cb2d910000:00cb2d977fff ~= 0022a000:00291fff
Device order is definitely sdf1, sde1, sdd1, sdc1, sdf1 ....
> End of /dev/sdc1 at 01d1c1016000
> End of /dev/sdd1 at 01d1c1016000
> End of /dev/sde1 at 01d1c0e90000
> End of /dev/sdf1 at 01d1c110e000
sdc1 and sdd1 are the same size and have the same chunk alignment
offsets. sdf1 is bigger but has the same chunk offsets as sdc1 and
sdd1. sde1 is smaller than sd[cd]1 by 0x186000 and has chunk offsets
0xf0000 earlier than the others for a given stripe.
sde1 has a data offset that ends in 0x10000 or 0x90000. 0x10000 is 64k,
an offset used by default early in mdadm's history.
The others have data offsets that end in 0x00000 or 0x80000. 0x80000 is
512k. 0x100000 is 1M, another default offset in mdadm history, and
happily the correct increment (0xf0000) bigger than the 64k suspicion I
have for sde1.
Here are the four --create operations I would try:
(This first one is most likely it based on the signature analysis in
November.)
mdadm --create --assume-clean --data-offset=variable \
--raid-devices=4 --chunk=512 --level=5 /dev/mdX \
/dev/sdc1:1024 /dev/sdf1:1024 /dev/sde1:64 /dev/sdd1:1024
mdadm --create --assume-clean --data-offset=variable \
--raid-devices=4 --chunk=512 --level=5 /dev/mdX \
/dev/sdf1:1024 /dev/sde1:64 /dev/sdd1:1024 /dev/sdc1:1024
mdadm --create --assume-clean --data-offset=variable \
--raid-devices=4 --chunk=512 --level=5 /dev/mdX \
/dev/sde1:64 /dev/sdd1:1024 /dev/sdc1:1024 /dev/sdf1:1024
mdadm --create --assume-clean --data-offset=variable \
--raid-devices=4 --chunk=512 --level=5 /dev/mdX \
/dev/sdd1:1024 /dev/sdc1:1024 /dev/sdf1:1024 /dev/sde1:64
Phil
^ permalink raw reply
* Re: broken raid level 5 array caused by user error
From: Mathias Mueller @ 2016-01-22 9:30 UTC (permalink / raw)
To: Phil Turmel; +Cc: Linux raid, linux-raid-owner
In-Reply-To: <15c75ec959427ae31d36912854b6debd@pingofdeath.de>
Hi Phil,
it's done:
658 Unique hashes in picture.jpg
Searching for pieces of picture.jpg in /dev/sdc1:...
/dev/sdc1 @ 005400d00000:005400d7ffff ~= 00156000:001d5fff
/dev/sdc1 @ 0087b4f29000:0087b4f7ffff ~= 00000000:00056fff
/dev/sdc1 @ 0087b4f80000:0087b4ffffff ~= 001d7000:00256fff
/dev/sdc1 @ 00cb2d980000:00cb2d9fffff ~= 0012a000:001a9fff
End of /dev/sdc1 at 01d1c1016000
Searching for pieces of picture.jpg in /dev/sdd1:...
/dev/sdd1 @ 005400c80000:005400cfffff ~= 000d6000:00155fff
/dev/sdd1 @ 0087b4f80000:0087b4ffffff ~= 00157000:001d6fff
/dev/sdd1 @ 00cb2d980000:00cb2d9fffff ~= 000aa000:00129fff
End of /dev/sdd1 at 01d1c1016000
Searching for pieces of picture.jpg in /dev/sde1:...
/dev/sde1 @ 005400b90000:005400c0ffff ~= 00056000:000d5fff
/dev/sde1 @ 005400c10000:005400c4bfff ~= 00256000:00291fff
/dev/sde1 @ 0087b4e10000:0087b4e8ffff ~= 000d7000:00156fff
/dev/sde1 @ 00cb2d810000:00cb2d88ffff ~= 0002a000:000a9fff
/dev/sde1 @ 00cb2d910000:00cb2d977fff ~= 0022a000:00291fff
End of /dev/sde1 at 01d1c0e90000
Searching for pieces of picture.jpg in /dev/sdf1:...
/dev/sdf1 @ 005400caa000:005400cfffff ~= 00000000:00055fff
/dev/sdf1 @ 005400d00000:005400d7ffff ~= 001d6000:00255fff
/dev/sdf1 @ 0087b4f00000:0087b4f7ffff ~= 00057000:000d6fff
/dev/sdf1 @ 0087b4f80000:0087b4fbafff ~= 00257000:00291fff
/dev/sdf1 @ 00cb2d956000:00cb2d97ffff ~= 00000000:00029fff
/dev/sdf1 @ 00cb2d980000:00cb2d9fffff ~= 001aa000:00229fff
End of /dev/sdf1 at 01d1c110e000
Mathias
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] raid6/algos.c : bug fix : Add the missing definitions to the pq.h file
From: Shaohua Li @ 2016-01-21 23:05 UTC (permalink / raw)
To: Gayatri Kammela
Cc: linux-raid, linux-kernel, NeilBrown, Anton Blanchard, Fenghua Yu
In-Reply-To: <1453413759-16064-1-git-send-email-gayatri.kammela@intel.com>
On Thu, Jan 21, 2016 at 02:02:39PM -0800, Gayatri Kammela wrote:
> Adding these pr_info and pr_err definitions so as to allow code to be
> compiled successfully for testing in userspace, since the printk has
> been replaced by pr_info and pr_err in algos.c
>
> Absence of these definitions result in the compilation errors
> such as ' undefined reference to `pr_info' ' ' undefined reference to
> `pr_err' '
>
> Cc: NeilBrown <neilb@suse.com>
> Cc: Anton Blanchard <anton@samba.org>
> Cc: Fenghua Yu <fenghua.yu@intel.com>
> Signed-off-by: Gayatri Kammela <gayatri.kammela@intel.com>
> ---
> include/linux/raid/pq.h | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/include/linux/raid/pq.h b/include/linux/raid/pq.h
> index a7a06d1dcf9c..a0118d5929a9 100644
> --- a/include/linux/raid/pq.h
> +++ b/include/linux/raid/pq.h
> @@ -152,6 +152,8 @@ void raid6_dual_recov(int disks, size_t bytes, int faila, int failb,
>
> # define jiffies raid6_jiffies()
> # define printk printf
> +# define pr_err(format, ...) fprintf(stderr, format, ## __VA_ARGS__)
> +# define pr_info(format, ...) fprintf(stdout, format, ## __VA_ARGS__)
> # define GFP_KERNEL 0
> # define __get_free_pages(x, y) ((unsigned long)mmap(NULL, PAGE_SIZE << (y), \
> PROT_READ|PROT_WRITE, \
Applied, thanks!
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox