* Re: RAID creation resync behaviors
From: Jes Sorensen @ 2017-05-09 21:22 UTC (permalink / raw)
To: Martin K. Petersen; +Cc: NeilBrown, Shaohua Li, linux-raid, neilb
In-Reply-To: <yq1bmr1rasn.fsf@oracle.com>
On 05/09/2017 05:16 PM, Martin K. Petersen wrote:
>
> Jes,
>
>> This is fine within the kernel, however it is not overly useful for
>> mdadm to determine which strategy to apply when syncing devices.
>
> BLKZEROOUT
>
Trying to read the code, as this ioctl doesn't seem to be documented
anywhere I can find.... it looks like this ioctl zeroes out a device.
It doesn't help me obtain the information I need to make a decision in
mdadm as whether to overwrite all or compare+write when resyncing a RAID
array.
Jes
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Phil Turmel @ 2017-05-09 21:23 UTC (permalink / raw)
To: Nix; +Cc: David Brown, Anthony Youngman, Ravi (Tom) Hale, linux-raid
In-Reply-To: <87mval4x6k.fsf@esperi.org.uk>
On 05/09/2017 04:01 PM, Nix wrote:
> On 9 May 2017, Phil Turmel told this:
>> The bottom line is that any kernel that implements the auto-correct you
>> seem to think is a slam dunk will be shunned by any system administrator
>> who actually cares about their data. Your obtuseness notwithstanding.
>
> Gee, thanks heaps. Next time I want randomly insulting by someone who
> doesn't bother to tell me his actual *arguments* in any message before
> the one that starts on the insults, I'll come straight to you.
Ok, yeah, I was a bit harsh. Ad hominem is not appropriate. Not that
the shunning wouldn't happen.
As for the arguments, well, *everyone* on this list is providing
arguments and you are ignoring them. Whether you are filtering facts on
pre-conceived ideas about raid6 or simply can't understand the points,
the result *appears* obtuse.
And now, please all drop me from the CC.
Phil
^ permalink raw reply
* Re: In Direct I/O, no matter how big the read/write, the bio's are all 4KB
From: Chris Worley @ 2017-05-09 21:26 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: linuxraid
In-Reply-To: <20170509201441.GA17309@infradead.org>
On Tue, May 9, 2017 at 10:14 AM, Christoph Hellwig <hch@infradead.org> wrote:
> On Tue, May 09, 2017 at 10:03:24AM -1000, Chris Worley wrote:
>> I realize DIO needs to be 4KB aligned,
>
> Actually many file systems do support sector size aligned direct I/O,
> which could be 512 byte aligned. But as that's not your point :)
>
>> but why do large I/Os get
>> chopped down into 4KB bio's?
>
> They shouldn't. What kernel version is this on?
$ uname -r
3.10.0-514.6.2.el7.x86_64
>
>> Doing a blocktrace of the md device, even though the app is writing at
>> 512KB, the MD driver is receiving 4KB bio's.
>
> The direct I/O code is using bio_add_page to build the bios it submits.
> It might be interesting to see which of the checks in it triggers
> an early return after the first page.
I vaguely recall the noop scheduler's only task is to merge adjacent
blocks. In my case, there's "none" (no schedular at all) for all
block devices chained together. Is that the issue?
Thanks,
Chris
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: NeilBrown @ 2017-05-09 21:32 UTC (permalink / raw)
To: Nix, Anthony Youngman; +Cc: Phil Turmel, Ravi (Tom) Hale, linux-raid
In-Reply-To: <87inla73vz.fsf@esperi.org.uk>
[-- Attachment #1: Type: text/plain, Size: 5958 bytes --]
On Tue, May 09 2017, Nix wrote:
> On 8 May 2017, Anthony Youngman told this:
>
>> If the scrub finds a mismatch, then the drives are reporting
>> "everything's fine here". Something's gone wrong, but the question is
>> what? If you've got a four-drive raid that reports a mismatch, how do
>> you know which of the four drives is corrupt? Doing an auto-correct
>> here risks doing even more damage. (I think a raid-6 could recover,
>> but raid-5 is toast ...)
>
> With a RAID-5 you are screwed: you can reconstruct the parity but cannot
> tell if it was actually right. You can make things consistent, but not
> correct.
>
> But with a RAID-6 you *do* have enough data to make things correct, with
> precisely the same probability as recovery of a RAID-5 "drive" of length
> a single sector. It seems wrong that not only does md not do this but
> doesn't even tell you which drive made the mistake so you could do the
> millions-of-times-slower process of a manual fail and readdition of the
> drive (or, if you suspect it of being wholly buggered, a manual fail and
> replacement).
>
>> And seeing as drives are pretty much guaranteed (unless something's
>> gone BADLY wrong) to either (a) accurately return the data written, or
>> (b) return a read error, that means a data mismatch indicates
>> something is seriously wrong that is NOTHING to do with the drives.
>
> This turns out not to be the case. See this ten-year-old paper:
> <https://indico.cern.ch/event/13797/contributions/1362288/attachments/115080/163419/Data_integrity_v3.pdf>.
> Five weeks of doing 2GiB writes on 3000 nodes once every two hours
> found, they estimated, 50 errors possibly attributable to disk problems
> (sector- or page-size regions of corrupted data) on 1/30th of their
> nodes. This is *not* rare and it is hard to imagine that 1/30th of disks
> used by CERN deserve discarding. It is better to assume that drives
> misdirect writes now and then, and to provide a means of recovering from
> them that does not take days of panic. RAID-6 gives you that means: md
> should use it.
>
> The page-sized regions of corrupted data were probably software -- but
> the sector-sized regions were just as likely the drives, possibly
> misdirected writes or misdirected reads.
>
> Neil decided not to do any repair work in this case on the grounds that
> if the drive is misdirecting one write it might misdirect the repair as
> well
My justification was a bit broader than that.
If you get a consistency error on RAID6, there is not one model to
explain it which is significantly more likely than any other model.
So it is not possible to predict the results of any particular remedial
action. It might help, it might hurt, it might have no effect.
Better to do nothing and appear incompetent, than to do the wrong thing
and remove all doubt.
(there could be problems with media, buffering in the drive, addressing
in the drive, buffer/addressing in the controller, errors in main
memory, CPU problems comparing bytes, corruption on a bus, either
reading or writing - of either data or addresses)
NeilBrown
> -- but if the repair is *consistently* misdirected, that seems
> relatively harmless (you had corruption before, you have it now, it just
> moved), and if it was a sporadic error, the repair is worthwhile. The
> only case in which a repair should not be attempted is if the drive is
> misdirecting all or most writes -- but in that case, by the time you do
> a scrub, on all but the quietest arrays you'll see millions of
> mismatches and it'll be obvious that it's time to throw the drive out.
> (Assuming md told you which drive it was.)
>
>>> If a sector weakens purely because of neighbouring writes or temperature
>>> or a vibrating housing or something (i.e. not because of actual damage),
>>> so that a rewrite will strengthen it and relocation was never necessary,
>>> surely you've just saved a pointless bit of sector sparing? (I don't
>>> know: I'm not sure what the relative frequency of these things is. Read
>>> and write errors in general are so rare that it's quite possible I'm
>>> worrying about nothing at all. I do know I forgot to scrub my old
>>> hardware RAID array for about three years and nothing bad happened...)
>>>
>> Yes you have saved a sector sparing. Note that a consumer 3TB drive
>> can return, on average, one error every time it's read from end to end
>> 3 times, and still be considered "within spec" ie "not faulty" by the
>
> Yeah, that's why RAID-6 is a good idea. :)
>
>> manufacturer. And that's a *brand* *new* drive. That's why building a
>> large array using consumer drives is a stupid idea - 4 x 3TB drives
>> and a *within* *spec* array must expect to handle at least one error
>> every scrub.
>
> That's just one reason why. The lack of control over URE timeouts is
> just as bad.
>
>> Okay - most drives are actually way over spec, and could probably be
>> read end-to-end many times without a single error, but you'd be a fool
>> to gamble on it.
>
> I'm trying *not* to gamble on it -- but I don't want to end up in the
> current situation we seem to have with md6, which is "oh, you have a
> mismatch, it's not going away, but we're neither going to tell you where
> it is nor what disk it's on nor repair it ourselves, even though we
> could, just to make it as hard as possible for you to repair the problem
> or even tell if it's a consistent one" (is the single mismatch an
> expected, spurious read error because of the volume of data you're
> reading, or one that's consistent and needs repair? All mismatch_cnt
> tells you is that there's a mismatch).
>
> --
> NULL && (void)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: RAID creation resync behaviors
From: Martin K. Petersen @ 2017-05-09 23:56 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Martin K. Petersen, NeilBrown, Shaohua Li, linux-raid, neilb
In-Reply-To: <f73c7524-5d35-6104-11b1-2c658b43c68d@gmail.com>
Jes,
>> BLKZEROOUT
>>
>
> Trying to read the code, as this ioctl doesn't seem to be documented
> anywhere I can find.... it looks like this ioctl zeroes out a device.
>
> It doesn't help me obtain the information I need to make a decision in
> mdadm as whether to overwrite all or compare+write when resyncing a RAID
> array.
I wasn't trying to solve your policy decision problem. I was merely
responding to Shaohua's concerns about discard vs. zeroes and wearing
out the media.
If you want to act based on the media type, the best heuristic we have
right now is the rotational sysfs attribute / BLKROTATIONAL ioctl. It'll
be one for spinning rust and zero for pretty much everything else.
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply
* Re: In Direct I/O, no matter how big the read/write, the bio's are all 4KB
From: Ming Lei @ 2017-05-10 2:47 UTC (permalink / raw)
To: Chris Worley; +Cc: Christoph Hellwig, linuxraid
In-Reply-To: <CANWz5fh_wkbug56gPwqH9pYO+hGdSgi+H-56bANpb8Q159zOfQ@mail.gmail.com>
On Wed, May 10, 2017 at 5:26 AM, Chris Worley <worleys@gmail.com> wrote:
> On Tue, May 9, 2017 at 10:14 AM, Christoph Hellwig <hch@infradead.org> wrote:
>> On Tue, May 09, 2017 at 10:03:24AM -1000, Chris Worley wrote:
>>> I realize DIO needs to be 4KB aligned,
>>
>> Actually many file systems do support sector size aligned direct I/O,
>> which could be 512 byte aligned. But as that's not your point :)
>>
>>> but why do large I/Os get
>>> chopped down into 4KB bio's?
>>
>> They shouldn't. What kernel version is this on?
>
> $ uname -r
> 3.10.0-514.6.2.el7.x86_64
>
>>
>>> Doing a blocktrace of the md device, even though the app is writing at
>>> 512KB, the MD driver is receiving 4KB bio's.
>>
>> The direct I/O code is using bio_add_page to build the bios it submits.
>> It might be interesting to see which of the checks in it triggers
>> an early return after the first page.
>
> I vaguely recall the noop scheduler's only task is to merge adjacent
> blocks. In my case, there's "none" (no schedular at all) for all
> block devices chained together. Is that the issue?
It shouldn't be related with scheduler since page is added to bio's bvec
table first. As Christoph mentioned, maybe some queue's limits are violated,
and pages from the 2nd one can't be added into any more. Since it might
depend on specific md setting, could you share your md setting or
detailed reproduction steps?
thanks,
Ming Lei
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Chris Murphy @ 2017-05-10 3:06 UTC (permalink / raw)
To: Wols Lists; +Cc: Chris Murphy, Nix, Phil Turmel, Ravi (Tom) Hale, Linux-RAID
In-Reply-To: <59120114.6080702@youngman.org.uk>
On Tue, May 9, 2017 at 11:49 AM, Wols Lists <antlists@youngman.org.uk> wrote:
> On 09/05/17 17:05, Chris Murphy wrote:
>>> Yes you have saved a sector sparing. Note that a consumer 3TB drive can
>>> > return, on average, one error every time it's read from end to end 3 times,
>>> > and still be considered "within spec" ie "not faulty" by the manufacturer.
>
>> All specs say "less than" which means it's a maximum permissible rate,
>> not an average. We have no idea what the minimum error rate is - we
>> being consumers. It's possible high volume users (e.g. Backblaze) have
>> data on this by now.
>>
> In other words, an error rate that high is "acceptable".
It's acceptable in that the manufacturer sells products with such
specification and consumers buy them. It's totally voluntary. There
are drives with one and two orders of magnitude lower unrecoverable
error rates and some people buy them and pay extra to get that spec as
a feature among other features.
> And to design software that quite explicitly expects greater perfection
> than the hardware itself is guaranteed to provide is, in my humble
> opinion, downright negligent!!!
How does the software expect a lower error rate than the drive specification?
--
Chris Murphy
^ permalink raw reply
* How to delay mdadm assembly until all component drives are recognized/ready?
From: Ram Ramesh @ 2017-05-10 3:13 UTC (permalink / raw)
To: Linux Raid
Today, I noticed that my RAID6 md0 was assembled in degraded state with
two drives in failed state after a pm-suspend and restart. Both of these
drives were attached toSAS9211-8I controller. The other drives are
attached to motherboard. I have not had this on a normal boot/reboot.
Also, in this particular case, mythtv recording was going on when
suspended and therefore as soon as resumed that used this md0.
Upon inspection, it appears (I am not sure here) that mdadm assembled
the array even before the drives were ready to be used. All I had to do
was to remove and re-add them to bring the array back to "good" state. I
am wondering if there is a way to tell mdadm to wait for all drives to
be ready before assembling. Also, if there is something that I can add
to resume scripts that will help, please let me know.
Kernel: Linux zym 3.13.0-106-generic #153-Ubuntu SMP
mdadm - v3.2.5 - 18th May 2012
Failed drives are HGST NAS and WD Gold with less than a year of usage.
So I doubt they are bad drives by any means.
Ramesh
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Chris Murphy @ 2017-05-10 3:53 UTC (permalink / raw)
To: Wols Lists; +Cc: Linux-RAID
In-Reply-To: <59121C1C.1050101@youngman.org.uk>
On Tue, May 9, 2017 at 1:44 PM, Wols Lists <antlists@youngman.org.uk> wrote:
>> This is totally non-trivial, especially because it says raid6 cannot
>> detect or correct more than one corruption, and ensuring that
>> additional corruption isn't introduced in the rare case is even more
>> non-trivial.
>
> And can I point out that that is just one person's opinion?
Right off the bat you ask a stupid question that contains the answer
to your own stupid question. This is condescending and annoying, and
it invites treating you with suspicious as a troll. But then you make
it worse by saying it again:
> A
> well-informed, respected person true, but it's still just opinion.
Except it is not just an opinion, it's a fact by any objective reader
who isn't even a programmer, let alone if you know something about
math and/or programming. Let's break down how totally stupid your
position is.
1. Opinions don't count for much.
2. You have presented no code that contradicts the opinion that this
is hard. You've opined that an opinion is to be discarded at face
value. Therefore your own opinion is just an opinion and likewise
discardable.
3. How do do the thing you think is trivial has been well documented
for some time and yet there are essentially no implementations. That
it's simple to do (your idea) and yet does not exist (fact) means this
is a big fat conspiracy to fuck you over, on purpose.
It's so asinine I feel trolled right now.
>And
> imho the argument that says raid should not repair the data applies
> equally against fsck - that shouldn't do any repair either! :-)
And now the dog shit cake has cat shit icing on it. Great.
>> And there is already something that will do exactly this: ZFS and
>> Btrfs. Both can unambiguously, efficiently determine whether data is
>> corrupt even if a drive doesn't report a read error.
>>
> Or we write an mdfsck program. Just like you shouldn't run fsck with
> write privileges on a mounted filesystem, you wouldn't run mdfsck with
> filesystems in the array mounted.
Who is we? Are you volunteering other people build you a feature?
> At the end of the day, md should never corrupt data by default. Which is
> what it sounds like is happening at the moment, if it's assuming the
> data sectors are correct and the parity is wrong. If one parity appears
> correct then by all means rewrite the second ...
This is an obtuse and frankly malicious characterization. Scrubs don't
happen by default. And scrub repair's assuming data strips are correct
is well documented. If you don't like this assumption, don't use scrub
repair. You can't say corruption happens by default unless you admit
that there's URE's on a drive by default - of course that's absurd and
makes no sense.
>
> But the current setup, where it's currently quite happy to assume a
> single-drive error and rewrite it if it's a parity drive, but it won't
> assume a single-drive error and and rewrite it if it's a data drive,
> just seems totally wrong. Worse, in the latter case, it seems it
> actively prevents fixing the problem by updating the parity and
> (probably) corrupting the data.
The data is already corrupted by definition. No additional damage to
data is done. What does happen is good P and Q are replaced by bad P
and Q which matches the already bad data.
And nevertheless you have the very real problem that drives lie about
having committed data to stable media. And they reorder writes,
breaking the write order assumptions of things. And we have RMW
happening on live arrays. And that means you have a real likelihood
that you cannot absolutely determine with the available information
why P and Q don't agree with the data, you're still making probability
assumptions and if that assumption is wrong any correction will
introduce more corruption.
The only unambiguous way to do this has already been done and it's ZFS
and Btrfs. And a big part of why they can do what they do is because
they are copy on write. IIf you need to solve the problem of ambiguous
data strip integrity in relation to P and Q, then use ZFS. It's
production ready. If you are prepared to help test and improve things,
then you can look into the Btrfs implementation.
Otherwise I'm sure md and LVM folks have a feature list that
represents a few years of work as it is without yet another pile on.
>
> Report the error, give the user the tools to fix it, and LET THEM sort
> it out. Just like we do when we run fsck on a filesystem.
They're not at all comparable. One is a file system, the other a raid
implementation, they have nothing in common.
--
Chris Murphy
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Wols Lists @ 2017-05-10 4:49 UTC (permalink / raw)
To: Chris Murphy; +Cc: Linux-RAID
In-Reply-To: <CAJCQCtR0y-g+8dKC2-fykmFnOPKMA5YQs-Hku67c79SMECQrEg@mail.gmail.com>
On 10/05/17 04:53, Chris Murphy wrote:
> On Tue, May 9, 2017 at 1:44 PM, Wols Lists <antlists@youngman.org.uk> wrote:
>
>>> This is totally non-trivial, especially because it says raid6 cannot
>>> detect or correct more than one corruption, and ensuring that
>>> additional corruption isn't introduced in the rare case is even more
>>> non-trivial.
>>
>> And can I point out that that is just one person's opinion?
>
> Right off the bat you ask a stupid question that contains the answer
> to your own stupid question. This is condescending and annoying, and
> it invites treating you with suspicious as a troll. But then you make
> it worse by saying it again:
>
Sorry. But I thought we were talking about *Neil's* paper. My bad for
missing it.
>> A
>> well-informed, respected person true, but it's still just opinion.
>
> Except it is not just an opinion, it's a fact by any objective reader
> who isn't even a programmer, let alone if you know something about
> math and/or programming. Let's break down how totally stupid your
> position is.
>
<snip ad hominems :-) >
>
>> At the end of the day, md should never corrupt data by default. Which is
>> what it sounds like is happening at the moment, if it's assuming the
>> data sectors are correct and the parity is wrong. If one parity appears
>> correct then by all means rewrite the second ...
>
> This is an obtuse and frankly malicious characterization. Scrubs don't
> happen by default. And scrub repair's assuming data strips are correct
> is well documented. If you don't like this assumption, don't use scrub
> repair. You can't say corruption happens by default unless you admit
> that there's URE's on a drive by default - of course that's absurd and
> makes no sense.
>
Documenting bad behaviour doesn't turn it into good behaviour, though ...
>>
>> But the current setup, where it's currently quite happy to assume a
>> single-drive error and rewrite it if it's a parity drive, but it won't
>> assume a single-drive error and and rewrite it if it's a data drive,
>> just seems totally wrong. Worse, in the latter case, it seems it
>> actively prevents fixing the problem by updating the parity and
>> (probably) corrupting the data.
>
> The data is already corrupted by definition. No additional damage to
> data is done. What does happen is good P and Q are replaced by bad P
> and Q which matches the already bad data.
Except, in my world, replacing good P & Q by bad P & Q *IS* doing
additional damage! We can identify and fix the bad data. So why don't
we? Throwing away good P & Q prevents us from doing that, and means we
can no longer recover the good data!
>
> And nevertheless you have the very real problem that drives lie about
> having committed data to stable media. And they reorder writes,
> breaking the write order assumptions of things. And we have RMW
> happening on live arrays. And that means you have a real likelihood
> that you cannot absolutely determine with the available information
> why P and Q don't agree with the data, you're still making probability
> assumptions and if that assumption is wrong any correction will
> introduce more corruption.
>
> The only unambiguous way to do this has already been done and it's ZFS
> and Btrfs. And a big part of why they can do what they do is because
> they are copy on write. IIf you need to solve the problem of ambiguous
> data strip integrity in relation to P and Q, then use ZFS. It's
> production ready. If you are prepared to help test and improve things,
> then you can look into the Btrfs implementation.
So how come btrfs and ZFS can handle this, and md can't? Can't md use
the same techniques. (Seriously, I don't know the answer. But, like Nix,
when I feel I'm being fed the answer "we're not going to give you the
choice because we know better than you", I get cheesed off. If I get the
answer "we're snowed under, do it yourself" then that is normal and
acceptable.)
>
> Otherwise I'm sure md and LVM folks have a feature list that
> represents a few years of work as it is without yet another pile on.
>
>>
>> Report the error, give the user the tools to fix it, and LET THEM sort
>> it out. Just like we do when we run fsck on a filesystem.
>
> They're not at all comparable. One is a file system, the other a raid
> implementation, they have nothing in common.
>
>
And what are file systems and raid implementations? They are both data
store abstractions. They have everything in common.
Oh and by the way, now I've realised my mistake, I've taken a look at
the paper you mention. In particular, section 4. Yes it does say you
can't detect and correct multi-disk errors - but that's not what we're
asking for!
By implication, it seems to be saying LOUD AND CLEAR that you CAN detect
and correct a single-disk error. So why the blankety-blank won't md let
you do that!
Neil's point seems to be that it's a bad idea to do it automatically. I
get his logic. But to then actively prevent you doing it manually - this
is the paternalistic attitude that gets my goat.
Anyways, I've been thinking about this, and I've got a proposal (RFC?).
I haven't got time right now - I'm supposed to be at work - but I'll
write it up this evening. If the response is "we're snowed under - it
sounds a good idea but do it yourself", then so be it. But if the
response is "we don't want the sysadmin to have the choice", then expect
more flak from people like Nix and me.
(And the proposal involves giving sysadmins CHOICE. If they want to take
the hit, it's *their* decision, not a paternalistic choice forced on them.)
(Sorry to keep on about paternalism, but there is a sense that decisions
have been made, and they're not going to be reversed "because I say so".
I'm NOT getting a "you want it, you write it" vibe, and that's what gets
to me.)
Cheers,
Wol
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Dave Stevens @ 2017-05-10 5:00 UTC (permalink / raw)
To: Chris Murphy; +Cc: Wols Lists, Linux-RAID
In-Reply-To: <CAJCQCtR0y-g+8dKC2-fykmFnOPKMA5YQs-Hku67c79SMECQrEg@mail.gmail.com>
Quoting Chris Murphy <lists@colorremedies.com>:
> On Tue, May 9, 2017 at 1:44 PM, Wols Lists <antlists@youngman.org.uk> wrote:
>
>>> This is totally non-trivial, especially because it says raid6 cannot
>>> detect or correct more than one corruption, and ensuring that
>>> additional corruption isn't introduced in the rare case is even more
>>> non-trivial.
>>
>> And can I point out that that is just one person's opinion?
>
> Right off the bat you ask a stupid question that contains the answer
snip!
you know Chris, I've read this twice and think it's abusive. You
shouldn't do this.
Dave
--
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."
-- John Dewey
^ permalink raw reply
* Re: RAID creation resync behaviors
From: Hannes Reinecke @ 2017-05-10 5:58 UTC (permalink / raw)
To: Jes Sorensen, Martin K. Petersen; +Cc: NeilBrown, Shaohua Li, linux-raid, neilb
In-Reply-To: <f73c7524-5d35-6104-11b1-2c658b43c68d@gmail.com>
On 05/09/2017 11:22 PM, Jes Sorensen wrote:
> On 05/09/2017 05:16 PM, Martin K. Petersen wrote:
>>
>> Jes,
>>
>>> This is fine within the kernel, however it is not overly useful for
>>> mdadm to determine which strategy to apply when syncing devices.
>>
>> BLKZEROOUT
>>
>
> Trying to read the code, as this ioctl doesn't seem to be documented
> anywhere I can find.... it looks like this ioctl zeroes out a device.
>
> It doesn't help me obtain the information I need to make a decision in
> mdadm as whether to overwrite all or compare+write when resyncing a RAID
> array.
>
What you actually want is the COMPARE AND WRITE SCSI command :-)
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
hare@suse.de +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: David Brown @ 2017-05-10 8:41 UTC (permalink / raw)
To: Nix, Chris Murphy
Cc: Anthony Youngman, Phil Turmel, Ravi (Tom) Hale, Linux-RAID
In-Reply-To: <87fugd4wdv.fsf@esperi.org.uk>
On 09/05/17 22:18, Nix wrote:
> On 9 May 2017, Chris Murphy verbalised:
>
>> On Tue, May 9, 2017 at 5:58 AM, David Brown <david.brown@hesbynett.no> wrote:
>>
>>> I thought you said that you had read Neil's article. Please go back and
>>> read it again. If you don't agree with what is written there, then
>>> there is little more I can say to convince you.
>
> The entire article is predicated on the assumption that when an
> inconsistent stripe is found, fixing it is simple because you can just
> fail whichever device is inconsistent... but given that the whole
> premise of the article is that *you cannot tell which that is*, I don't
> see the point in failing anything.
The point is that if an inconsistent stripe is found, then there is no
way to be sure how to fix it correctly. So scrub certainly will not
touch it. And what should "repair" do? I see several choices:
1. It could assume the data is correct, and re-create the parities.
This is simple, and it avoids changing anything on the array from the
viewpoint of higher levels (i.e., the filesystem).
2. It could do a "smart" repair of the stripe, if it sees that there is
only one inconsistent block in the stripe.
3. It could pass the problem on to higher level tools (possibly
correcting a single inconsistency in the P or Q parities first).
At the moment, raid6 repair follows the first choice here. Many people
seem to think the second choice is a good idea. Personally, I would say
choice 3 is right - but unless and until higher level tools are
available, I think 1 is no worse than 2 - and it is simpler, clearer,
and works today.
Key to why I don't like choice 2 is a question of why you have a
mismatch in the first place. Undetected read errors - the drive
returning wrong data as though it were correct data - are astoundingly
rare. Even on huge disks, they do not occur often. (Unrecoverable read
errors - when the drive reports a sector as unreadable - are not
uncommon. That is what raid is for.) If you get a mismatch, a likely
cause is a crash or power fault during a stripe write. Another main
cause is hardware errors such as memory faults. "Smart" repair can make
the situation worse.
Secondly, "smart" repair means changing the data on the disk. You can't
do that while a file system is mounted (unless you want to risk chaos).
One major reason for using raid is to minimise downtime of a system in
the event of problems - offline repair goes against that philosophy.
What do I mean about passing the problem on to higher levels? One
example would be if there is an other raid level sitting above, such as
a raid1 pair of raid6 arrays (it would make more sense the other way
round - the same principle applies there). The raid6 level could ask
the block layer above if that layer can re-create the correct data. In
the case of a raid1 pair at a higher level, then it could - that way the
stripe would be written with the full known correct data, rather than
just a guess. Perhaps the layer above is a filesystem - this could say
if that stripe is actually in use (no need to worry if it is in deleted
space), or if it can re-create the data from a BTRFS duplicate.
Failing that, a tool could interact with the filesystem to determine
what sort of data was on that stripe, and perhaps check it in some way.
At least a tool could run a consistency check - would the filesystem be
consistent if the stripe was "smart repaired", or would it be consistent
if the stripe data was left untouched (and the P & Q parities recreated)?
A simple method here could be to mark the whole stripe as unreadable,
then run a filesystem check. If there are higher level raids that can
re-create the lost stripe, that will happen automatically. If not, then
the filesystem repair will ensure that the filesystem is consistent even
though data may be lost.
And of course, a higher level repair tool could be one that simply runs
a "smart repair" on the stripe.
All in all, when there is /no/ correct answer, I think we have to be
very careful about picking methods here. Before switching to a "smart"
repair, rather than the simple method, we have to be /very/ sure that it
gives noticeably "better" results in real-world cases. We can't just
say it sounds good - we need to know.
>
> The first comment in the article is someone noting that md doesn't say
> which device is failing, what the location of the error is or anything
> else a sysadmin might actually find useful for fixing it. "Hey, you have
> an error somewhere on some disk on this multi-terabyte array which might
> be data corruption and if a disk fails will be data corruption!" is not
> too useful :(
I haven't looked at the information you get out of the scrub, but of
course more information is better than less information.
> The fourth comment notes that the "smart" approach, given
> RAID-6, has a significantly higher chance of actually fixing the problem
> than the simple approach. I'd call that a fairly important comment...
>
> (Neil said: "Similarly a RAID6 with inconsistent P and Q could well not
> be able to identify a single block which is "wrong" and even if it could
> there is a small possibility that the identified block isn't wrong, but
> the other blocks are all inconsistent in such a way as to accidentally
> point to it. The probability of this is rather small, but it is
> non-zero".
It is true that for some causes of mismatches, the "smart" repair has a
high chance of being correct.
> As far as I can tell the probability of this is exactly the
> same as that of multiple read errors in a single stripe -- possibly far
> lower, if you need not only multiple wrong P and Q values but *precisely
> mis-chosen* ones. If that wasn't acceptably rare, you wouldn't be using
> RAID-6 to begin with.
>
> I've been talking all the time about a stripe which is singly
> inconsistent: either all the data blocks are fine and one of P or Q is
> fine, or both P and Q and all but one data block is fine, and the
> remaining block is inconsistent with all the rest. Obviously if more
> blocks are corrupt, you can do nothing but report it. The redundancy
> simply isn't there to attempt repair.)
Or possible mark the whole stripe as "unreadable", and punt the problem
to the higher levels.
>
>> H. Peter Anvin's RAID 6 paper, section 4 is what's apparently under discussion
>> http://milbret.anydns.info/pub/linux/kernel/people/hpa/raid6.pdf
>>
>> This is totally non-trivial, especially because it says raid6 cannot
>> detect or correct more than one corruption, and ensuring that
>> additional corruption isn't introduced in the rare case is even more
>> non-trivial.
>
> Yeah. Testing this is the bastard problem, really. Fault injection via
> dm is the only approach that seems remotely practical to me.
That's what the "FAULTY" raid level in md is for :-)
But what are the /realistic/ fault situations?
>
>> I do think it's sane for raid6 repair to avoid the current assumption
>> that data strip is correct, by doing the evaluation in equation 27. If
>> there's no corruption do nothing, if there's corruption of P or Q then
>> replace, if there's corruption of data, then report but do not repair
>
> At least indicate *where* the corruption is in the report. (I'd say
> "repair, as a non-default option" for people with a different
> availability/P(corruption) tradeoff -- since, after all, if you're using
> RAID In the first place you value high availability across disk problems
> more than most people do, and there is a difference between one bit of
> unreported damage that causes a near-certain restore from backup and
> either zero or two of them plus a report with an LBA attached so you
> know you need to do something...)
One thing to consider here is the sort of person using the raid array.
When Neil wrote his article, raid6 would only be used by an expert. He
did not want to change existing data and make life harder for the
systems administrator doing more serious repair.
However, these days the raid6 "administrator" may be someone who owns a
NAS box and has no idea what raid, or even Linux, actually is. In such
cases, "smart" repair is probably the best idea if the filesystem on top
is not BTRFS.
>
>> as follows:
>>
>> 1. md reports all data drives and the LBAs for the affected stripe
>> (otherwise this is not simple if it has to figure out which drive is
>> actually affected but that's not required, just a matter of better
>> efficiency in finding out what's really affected.)
>
> Yep.
>
>> 2. the file system needs to be able to accept the error from md
>
> It would probably need to report this as an -EIO, but I don't know of
> any filesystems that can accept asynchronous reports of errors like
> this. You'd need reverse mapping to even stand a chance (a non-default
> option on xfs, and of course available on btrfs and zfs too). You'd
> need self-healing metadata to stand a chance of doing anything about it.
> And god knows what a filesystem is meant to do if part of the file data
> vanishes. Replace it with \0? ugh. I'd almost rather have the error
> go back out to a monitoring daemon and have it send you an email...
>
>> 3. the file system reports what it negatively impacted: file system
>> metadata or data and if data, the full filename path.
>>
>> And now suddenly this work is likewise non-trivial.
>
> Yeah, it's all the layers stacked up to the filesystem that are buggers
> to deal with... and now the optional 'just repair it dammit' approach
> seems useful again, if just because it doesn't have to deal with all
> these extra layers.
>
>> And there is already something that will do exactly this: ZFS and
>> Btrfs. Both can unambiguously, efficiently determine whether data is
>> corrupt even if a drive doesn't report a read error.
>
> Yeah. Unfortunately both have their own problems: ZFS reimplements the
> page cache and adds massive amounts of ineffiicency in the process, and
> btrfs is... well... not really baked enough for the sort of high-
> availability system that's going to be running RAID, yet. (Alas!)
I disagree about BTRFS here. First, raid is a good idea no matter how
"experimental" you consider your filesystem. Second, BTRFS is solid
enough for a great many uses - I use it on laptops, desktops and servers.
/No/ storage system should be viewed as infallible - backups are
important. So if BTRFS were to eat my data, then I'd get it back from
backups - just as I would if the server died, both disks failed, it got
stolen, or whatever.
But BTRFS on our servers means very cheap regular snapshots. That
protects us from the biggest cause of data loss - user error.
>
> (Recent xfs can do the same with metadata, but not data.)
>
^ permalink raw reply
* Network based (iSCSI) RAID1 setup
From: Gionatan Danti @ 2017-05-10 8:42 UTC (permalink / raw)
To: linux-raid
Hi all,
I'm trying to understand if, and how, mdadm can be used with network
attached devices (iSCSI, in this case). I have a very simple setup with
two 1 GB drives, the first being a local disk (a logical volume, really)
and the second a remote iSCSI disk.
First question: even if in my preliminary tests this seems to work
reasonably well, do you feel that such solution can be used for
production workloads? Or something with a more specific focus, as DRBD,
remains the preferred solution?
I'm using two CentOS 7.3 x86-64 boxes, with kernel version
3.10.0-514.16.1.el7.x86_64 and mdadm v3.4 - 28th January 2016. Here you
can find my current RAID1 setup, where /dev/sdb is the iSCSI disk:
[root@gdanti-laptop g.danti]# cat /proc/mdstat
Personalities : [raid1]
md200 : active raid1 sdb[1] dm-3[0]
1047552 blocks super 1.2 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk
unused devices: <none>
[root@gdanti-laptop g.danti]# mdadm -D /dev/md200
/dev/md200:
Version : 1.2
Creation Time : Wed May 10 08:53:12 2017
Raid Level : raid1
Array Size : 1047552 (1023.00 MiB 1072.69 MB)
Used Dev Size : 1047552 (1023.00 MiB 1072.69 MB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed May 10 10:27:35 2017
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Name : gdanti-laptop.assyoma.it:200 (local to host
gdanti-laptop.assyoma.it)
UUID : 9d6fb056:c1d49780:149f9391:68fc267f
Events : 62
Number Major Minor RaidDevice State
0 253 3 0 active sync /dev/dm-3
1 8 16 1 active sync /dev/sdb
So far, so good: the array seems to work well, with good read/write
speed. Now, suppose the remote disk become unavailable:
[root@gdanti-laptop g.danti]# iscsiadm -m node --targetname
iqn.2008-09.com.example:server.target1 --portal 172.31.255.11 --logout
Logging out of session [sid: 6, target:
iqn.2008-09.com.example:server.target1, portal: 172.31.255.11,3260]
Logout of [sid: 6, target: iqn.2008-09.com.example:server.target1,
portal: 172.31.255.11,3260] successful.
[root@gdanti-laptop g.danti]# cat /proc/mdstat
Personalities : [raid1]
md200 : active (auto-read-only) raid1 dm-3[0]
1047552 blocks super 1.2 [2/1] [U_]
bitmap: 0/1 pages [0KB], 65536KB chunk
unused devices: <none>
The device is correctly kicked-off the array.
So, second question: how to enable auto re-add for the remote device
when it become available again? For example:
[root@gdanti-laptop g.danti]# iscsiadm -m node --targetname
iqn.2008-09.com.example:server.target1 --portal 172.31.255.11 --login
Logging in to [iface: default, target:
iqn.2008-09.com.example:server.target1, portal: 172.31.255.11,3260]
(multiple)
Login to [iface: default, target:
iqn.2008-09.com.example:server.target1, portal: 172.31.255.11,3260]
successful.
[root@gdanti-laptop g.danti]# cat /proc/mdstat
Personalities : [raid1]
md200 : active (auto-read-only) raid1 dm-3[0]
1047552 blocks super 1.2 [2/1] [U_]
bitmap: 0/1 pages [0KB], 65536KB chunk
unused devices: <none>
Even if /dev/sdb is now visible, it is not auto re-added to the array.
If I run mdadm /dev/sdb --incremental --run I see the device added as a
spare:
[root@gdanti-laptop g.danti]# cat /proc/mdstat
Personalities : [raid1]
md200 : active (auto-read-only) raid1 sdb[1](S) dm-3[0]
1047552 blocks super 1.2 [2/1] [U_]
bitmap: 0/1 pages [0KB], 65536KB chunk
unused devices: <none>
Third question: with --incremental adds device as a spare, rather than
active?
I've looked at the POLICY directive in mdadm.conf, but I am unable to
make it work by auto re-adding iSCSI devices when they become up again.
Sorry for the long post, I am really trying to learn something!
Thanks.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8
^ permalink raw reply
* Re: [PATCH] IMSM: Correct --examine output for 4k disks
From: Mariusz Dabrowski @ 2017-05-10 8:43 UTC (permalink / raw)
To: Jes Sorensen, linux-raid; +Cc: Maksymilian Kunt
In-Reply-To: <1aca3793-3786-00dc-910e-920a21a1108b@gmail.com>
On 05/09/2017 06:04 PM, Jes Sorensen wrote:
> On 05/09/2017 08:03 AM, Mariusz Dabrowski wrote:
>> From: Maksymilian Kunt <maksymilian.kunt@intel.com>
>>
>> "Array Size" and "Per Dev Size" are incorrect for disks with sector size
>> different than 512B.
>>
>> Calculate "Array Size" and "Per Dev Size" based on sector size. Additionally
>> print "Sector Size".
>>
>> Signed-off-by: Maksymilian Kunt <maksymilian.kunt@intel.com>
>> Signed-off-by: Mariusz Dabrowski <mariusz.dabrowski@intel.com>
>> ---
>> super-intel.c | 7 +++++--
>> 1 file changed, 5 insertions(+), 2 deletions(-)
>
> I am a little wary of this change as it changes the order of the output. Does
> anyone have tools parsing this information which could break from this?
>
> Jes
>
Hi Jes,
some time ago we have added "RWH Policy" field to --examine output and we
haven't heard any complaints about that change.
Also it is better to parse output from --export option which is unchanged in
this patch.
Thanks,
Mariusz
^ permalink raw reply
* Re: Network based (iSCSI) RAID1 setup
From: Roman Mamedov @ 2017-05-10 9:03 UTC (permalink / raw)
To: Gionatan Danti; +Cc: linux-raid
In-Reply-To: <dc2c3583-14da-435b-0177-8cee749a1cd0@assyoma.it>
On Wed, 10 May 2017 10:42:54 +0200
Gionatan Danti <g.danti@assyoma.it> wrote:
> even if in my preliminary tests this seems to work
> reasonably well, do you feel that such solution can be used for
> production workloads? Or something with a more specific focus, as DRBD,
> remains the preferred solution?
First thing that comes to mind, you should look into setting the remote device
as --write-mostly, so that the local one is preferred for all reads (as long as
it's up).
But to be honest DRBD may indeed be a better solution for this use case, as
it's built specifically with it in mind, and likely has all the various
gotchas that might arise already thought about and handled properly.
--
With respect,
Roman
^ permalink raw reply
* [PATCH] dm ioctl: make dm_open and dm_release release
From: Colin King @ 2017-05-10 9:18 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, dm-devel, Shaohua Li, linux-raid
Cc: kernel-janitors, linux-kernel
From: Colin Ian King <colin.king@canonical.com>
Making dm_open and dm_release static fixes the sparse warnings:
warning: symbol 'dm_open' was not declared. Should it be static?
warning: symbol 'dm_release' was not declared. Should it be static?
Fixes: ab2c6224f4c0c ("dm: a basic support for using the select or poll function")
Signed-off-by: Colin Ian King <colin.king@canonical.com>
---
drivers/md/dm-ioctl.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/md/dm-ioctl.c b/drivers/md/dm-ioctl.c
index 70dc5db90ef2..d45c68115d4c 100644
--- a/drivers/md/dm-ioctl.c
+++ b/drivers/md/dm-ioctl.c
@@ -1893,7 +1893,7 @@ static long dm_compat_ctl_ioctl(struct file *file, uint command, ulong u)
#define dm_compat_ctl_ioctl NULL
#endif
-int dm_open(struct inode *inode, struct file *filp)
+static int dm_open(struct inode *inode, struct file *filp)
{
int r;
struct dm_file *priv;
@@ -1911,7 +1911,7 @@ int dm_open(struct inode *inode, struct file *filp)
return 0;
}
-int dm_release(struct inode *inode, struct file *filp)
+static int dm_release(struct inode *inode, struct file *filp)
{
kfree(filp->private_data);
return 0;
--
2.11.0
^ permalink raw reply related
* [PATCH][V2] dm ioctl: make dm_open and dm_release release static
From: Colin King @ 2017-05-10 9:18 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, dm-devel, Shaohua Li, linux-raid
Cc: kernel-janitors, linux-kernel
From: Colin Ian King <colin.king@canonical.com>
Making dm_open and dm_release static fixes the sparse warnings:
warning: symbol 'dm_open' was not declared. Should it be static?
warning: symbol 'dm_release' was not declared. Should it be static?
Fixes: ab2c6224f4c0c ("dm: a basic support for using the select or poll function")
Signed-off-by: Colin Ian King <colin.king@canonical.com>
---
drivers/md/dm-ioctl.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/md/dm-ioctl.c b/drivers/md/dm-ioctl.c
index 70dc5db90ef2..d45c68115d4c 100644
--- a/drivers/md/dm-ioctl.c
+++ b/drivers/md/dm-ioctl.c
@@ -1893,7 +1893,7 @@ static long dm_compat_ctl_ioctl(struct file *file, uint command, ulong u)
#define dm_compat_ctl_ioctl NULL
#endif
-int dm_open(struct inode *inode, struct file *filp)
+static int dm_open(struct inode *inode, struct file *filp)
{
int r;
struct dm_file *priv;
@@ -1911,7 +1911,7 @@ int dm_open(struct inode *inode, struct file *filp)
return 0;
}
-int dm_release(struct inode *inode, struct file *filp)
+static int dm_release(struct inode *inode, struct file *filp)
{
kfree(filp->private_data);
return 0;
--
2.11.0
^ permalink raw reply related
* Re: [PATCH] dm ioctl: make dm_open and dm_release release
From: Colin Ian King @ 2017-05-10 9:19 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, dm-devel, Shaohua Li, linux-raid
Cc: kernel-janitors, linux-kernel
In-Reply-To: <20170510091821.17995-1-colin.king@canonical.com>
On 10/05/17 10:18, Colin King wrote:
> From: Colin Ian King <colin.king@canonical.com>
>
> Making dm_open and dm_release static fixes the sparse warnings:
>
> warning: symbol 'dm_open' was not declared. Should it be static?
> warning: symbol 'dm_release' was not declared. Should it be static?
>
> Fixes: ab2c6224f4c0c ("dm: a basic support for using the select or poll function")
> Signed-off-by: Colin Ian King <colin.king@canonical.com>
> ---
> drivers/md/dm-ioctl.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/md/dm-ioctl.c b/drivers/md/dm-ioctl.c
> index 70dc5db90ef2..d45c68115d4c 100644
> --- a/drivers/md/dm-ioctl.c
> +++ b/drivers/md/dm-ioctl.c
> @@ -1893,7 +1893,7 @@ static long dm_compat_ctl_ioctl(struct file *file, uint command, ulong u)
> #define dm_compat_ctl_ioctl NULL
> #endif
>
> -int dm_open(struct inode *inode, struct file *filp)
> +static int dm_open(struct inode *inode, struct file *filp)
> {
> int r;
> struct dm_file *priv;
> @@ -1911,7 +1911,7 @@ int dm_open(struct inode *inode, struct file *filp)
> return 0;
> }
>
> -int dm_release(struct inode *inode, struct file *filp)
> +static int dm_release(struct inode *inode, struct file *filp)
> {
> kfree(filp->private_data);
> return 0;
>
Sorry, ignore this, it had a mistake in the subject line
^ permalink raw reply
* Re: Network based (iSCSI) RAID1 setup
From: Gionatan Danti @ 2017-05-10 10:25 UTC (permalink / raw)
To: Roman Mamedov; +Cc: linux-raid
In-Reply-To: <20170510140323.256edb95@natsu>
On 10/05/2017 11:03, Roman Mamedov wrote:
> First thing that comes to mind, you should look into setting the remote device
> as --write-mostly, so that the local one is preferred for all reads (as long as
> it's up).
Sure, and it is planned. However, for initial testing, I want to leave
as many parameters to their default settings.
> But to be honest DRBD may indeed be a better solution for this use case, as
> it's built specifically with it in mind, and likely has all the various
> gotchas that might arise already thought about and handled properly.
>
To tell the truth, I already use DRBD 8.4 in production workloads and I
are quite satisfied with it. However, DRBD 8.4 only supports 2 hosts
(ie: master and slave), and DRBD 9.x (which supports multi-node
replication) is a relatively new, deep re-write of the old codebase
which significantly expanded scope.
So my interest in mdadm-based network replication...
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti@assyoma.it - info@assyoma.it
GPG public key ID: FF5F32A8
^ permalink raw reply
* [PATCH][HSD116699] raid1: prefer disk without bad blocks
From: Tomasz Majchrzak @ 2017-05-10 10:53 UTC (permalink / raw)
To: linux-raid; +Cc: shli, Tomasz Majchrzak
If an array consists of two drives and the first drive has the bad
block, the read request to the region overlapping the bad block chooses
the same disk (with bad block) as device to read from over and over and
the request gets stuck. If the first disk only partially overlaps with
bad block, it becomes a candidate ("best disk") for shorter range of
sectors. The second disk is capable of reading the entire requested
range and it is updated accordingly, however it is not recorded as a
best device for the request. In the end the request is sent to the first
disk to read entire range of sectors. It fails and is re-tried in a
moment but with the same outcome.
Actually it is quite likely scenario but it had little exposure in my
test until commit 715d40b93b10 ("md/raid1: add failfast handling for
reads.") removed preference for idle disk. Such scenario had been passing
as second disk was always chosen when idle.
Update a candidate ("best disk") to read from if disk can read entire
range. Do it only if other disk has already been chosen as a candidate
for a smaller range. Otherwise, don't update it - let the head position /
disk type logic to select a disk.
Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
---
drivers/md/raid1.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index a1f3fbe..e7ab3bb 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -628,8 +628,11 @@ static int read_balance(struct r1conf *conf, struct r1bio *r1_bio, int *max_sect
break;
}
continue;
- } else
+ } else {
+ if ((sectors > best_good_sectors) && (best_disk >= 0))
+ best_disk = disk;
best_good_sectors = sectors;
+ }
if (best_disk >= 0)
/* At least two disks to choose from so failfast is OK */
--
1.8.3.1
^ permalink raw reply related
* Re: Network based (iSCSI) RAID1 setup
From: Adam Goryachev @ 2017-05-10 12:08 UTC (permalink / raw)
To: Gionatan Danti, linux-raid
In-Reply-To: <dc2c3583-14da-435b-0177-8cee749a1cd0@assyoma.it>
On 10/5/17 18:42, Gionatan Danti wrote:
> Hi all,
> I'm trying to understand if, and how, mdadm can be used with network
> attached devices (iSCSI, in this case). I have a very simple setup
> with two 1 GB drives, the first being a local disk (a logical volume,
> really) and the second a remote iSCSI disk.
>
> First question: even if in my preliminary tests this seems to work
> reasonably well, do you feel that such solution can be used for
> production workloads? Or something with a more specific focus, as
> DRBD, remains the preferred solution?
>
It depends on your definition of production, but for me, the answer is
no. Once upon a time, I used MD to do RAID1 between a local SSD and a
remote device with NBD and that worked well, (apart from the fact I
needed to manually re-add the remote device after a reboot, or whenever
it dropped out for any other reason). It did save me when the local SSD
died, and I was able to keep running purely from the remote NBD device
until I could get in and replace the local SSD.
Today, I use DRBD, and would much prefer that compared to MD + NBD.
> I'm using two CentOS 7.3 x86-64 boxes, with kernel version
> 3.10.0-514.16.1.el7.x86_64 and mdadm v3.4 - 28th January 2016. Here
> you can find my current RAID1 setup, where /dev/sdb is the iSCSI disk:
>
> So, second question: how to enable auto re-add for the remote device
> when it become available again? For example:
I don't know, but I guess you need to work out what udev rules are
triggered when the iscsi device is "connected", and then get that to
trigger the MD add rules. Possibly you could try to create a partition
on the iscsi, and then use sdb1 for the RAID array, there might be
better handling by udev in that case (I really don't know, just making
random suggestions here).
> Even if /dev/sdb is now visible, it is not auto re-added to the array.
> If I run mdadm /dev/sdb --incremental --run I see the device added as
> a spare:
>
> [root@gdanti-laptop g.danti]# cat /proc/mdstat
> Personalities : [raid1]
> md200 : active (auto-read-only) raid1 sdb[1](S) dm-3[0]
> 1047552 blocks super 1.2 [2/1] [U_]
> bitmap: 0/1 pages [0KB], 65536KB chunk
>
> unused devices: <none>
>
> Third question: with --incremental adds device as a spare, rather than
> active?
>
Is it because the raid isn't actually running? Perhaps you need to start
the array first?
> I've looked at the POLICY directive in mdadm.conf, but I am unable to
> make it work by auto re-adding iSCSI devices when they become up again.
I'd suggest using DRBD, it handles all these things a lot better because
it is normal events for it, and a lot more people will be able to assist
when something goes wrong with it.
Regards,
Adam
^ permalink raw reply
* [PATCH v2] raid1: prefer disk without bad blocks
From: Tomasz Majchrzak @ 2017-05-10 13:16 UTC (permalink / raw)
To: linux-raid; +Cc: shli, Tomasz Majchrzak
In-Reply-To: <1494413599-8749-1-git-send-email-tomasz.majchrzak@intel.com>
If an array consists of two drives and the first drive has the bad
block, the read request to the region overlapping the bad block chooses
the same disk (with bad block) as device to read from over and over and
the request gets stuck. If the first disk only partially overlaps with
bad block, it becomes a candidate ("best disk") for shorter range of
sectors. The second disk is capable of reading the entire requested
range and it is updated accordingly, however it is not recorded as a
best device for the request. In the end the request is sent to the first
disk to read entire range of sectors. It fails and is re-tried in a
moment but with the same outcome.
Actually it is quite likely scenario but it had little exposure in my
test until commit 715d40b93b10 ("md/raid1: add failfast handling for
reads.") removed preference for idle disk. Such scenario had been passing
as second disk was always chosen when idle.
Update a candidate ("best disk") to read from if disk can read entire
range. Do it only if other disk has already been chosen as a candidate
for a smaller range. Otherwise, don't update it - let the head position /
disk type logic to select a disk.
Signed-off-by: Tomasz Majchrzak <tomasz.majchrzak@intel.com>
---
drivers/md/raid1.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
v2:
updated the title not to include internal defect number :)
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index a1f3fbe..e7ab3bb 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -628,8 +628,11 @@ static int read_balance(struct r1conf *conf, struct r1bio *r1_bio, int *max_sect
break;
}
continue;
- } else
+ } else {
+ if ((sectors > best_good_sectors) && (best_disk >= 0))
+ best_disk = disk;
best_good_sectors = sectors;
+ }
if (best_disk >= 0)
/* At least two disks to choose from so failfast is OK */
--
1.8.3.1
^ permalink raw reply related
* RFC - Raid error detection and auto-recovery (was Fault tolerance with badblocks)
From: Wols Lists @ 2017-05-10 13:26 UTC (permalink / raw)
To: linux-raid, Nix
This discussion seems to have become a bit heated, but I think we have
the following:
FACT: linux md raid can do error detection but doesn't. Why not? It
seems people are worried about the performance hit.
FACT: linux md raid can do automatic error correction but doesn't. Why
not? It seems people are more worried about the problems it could cause
than the problems it would fix.
OBSERVATION: The kernel guys seem to get fixated on kernel performance
and miss the bigger picture. At the end of the day, the most important
thing on the computer is the USER'S DATA. And if we can't protect that,
they'll throw the computer in the bin. Or replace linux with Windows. Or
something like that. And when there's a problem, it all too often comes
over that the kernel guys CAN fix it but WON'T. The ext2/3/4 transition
is a case in point. The current frustration where the kernel guys say
"user data is the application's problem" but the postgresql guys are
saying "how can we guarantee integrity when you won't give us the tools
we need to guarantee our data is safe".
This situation smacks of the same arrogance, sorry. "We can save your
data but we won't".
FURTHER FACTUAL TIDBITS:
The usual response seems to be to push the problem somewhere else. For
example "The user should keep backups". BUT HOW? I've investigated!
Let's say I buy a spare drive for my backup. But I installed raid to
avoid being at the mercy of a single drive. Now I am again because my
backup is a single drive! BIG FAIL.
Okay, I'll buy two drives, and have a backup raid. But what if my backup
raid is reporting a mismatch count too? Now I have TWO copies where I
can't vouch for their integrity. Double the trouble. BIG FAIL.
Tape is cheap, you say? No bl***ding way!!! I've just done a quick
investigation, and for the price of a tape drive I could probably turn
my 2x3TB raid-1 into a 3x3TB raid-5, AND buy sufficient disks to
implement a raid-based grandfather/father/son backup procedure, and
STILL have some change left over. (I am using cheapie desktop drives,
but I could probably afford cheap NAS drives with that money.)
PROPOSAL: Enable integrity checking.
We need to create something like /sys/md/array/verify_data_on_read. If
that's set to true and we can check integrity (ie not raid-0), rather
than reading just the data disks, we read the entire stripe, check the
mirror or parity, and then decide what to do. If we can return
error-corrected data obviously we do. I think we should return an error
if we can't, no?
We can't set this by default. The *potential* performance hit is too
great. But now the sysadmin can choose between performance or integrity,
rather than the present state where he has no choice. And in reality, I
don't think a system like mine would even notice! Low read/write
activity, and masses of spare ram. Chances are most of my disk activity
is cached and doesn't go anywhere near the raid code.
The kernel code size impact is minimal, I suspect. All the code required
is probably there, it just needs a little "re-purposing".
PROPOSAL: Enable automatic correction
Likewise create /sys/md/array/correct_data_on_read. This won't work if
verify_data_on_read is not set, and likewise it will not be set by
default. IFF we need to reconstruct the data from a 3-or-more raid-1
mirror or a raid-6, it will rewrite the corrected stripe.
RATIONALE:
NEVER THROW AWAY USER DATA IF YOU CAN RECONSTRUCT IT !!!
This gives control to the sysadmin. At the end of the day, it should be
*his* call, not the devs', as to whether verify-on-read is worth the
performance hit. (Successful reconstructions should be logged ...)
Likewise, while correct_data_on_read could mess up the array if the
error isn't actually on the drive, that should be the sysadmin's call,
not the devs'. And because we only rewrite if we think we have
successfully recreated the data, the chances of it messing up are
actually quite small. Because verify_data_on_read is set, that addresses
Neil's concern of changing the data underneath an app - the app has been
given the corrected data so we write the corrected data back to disk.
NOTES:
From Peter Anvin's paper it seems that the chance of wrongly identifying
a single-disk error is low. And it's even lower if we look for the clues
he mentions. Because we only correct those errors we are sure we've
correctly identified, other sources of corruption shouldn't get fed back
to the disk.
This makes an error-correcting scrub easy :-) Run as an overnight script...
cat 1 > /sys/md/root/verify_data_on_read
cat 1 > /sys/md/root/correct_data_on_read
tar -c / > /dev/null
cat 0 > /sys/md/root/correct_data_on_read
cat 0 > /sys/md/root/verify_data_on_read
Coders and code welcome ... :-)
Cheers,
Wol
^ permalink raw reply
* Resync of the degraded RAID10 array
From: Tomasz Majchrzak @ 2017-05-10 14:00 UTC (permalink / raw)
To: linux-raid
Hi all,
I wonder what should be the resync behaviour for the degraded RAID10 array.
cat /proc/mdstat
Personalities : [raid10]
md127 : active raid10 nvme3n1[3] nvme2n1[2] nvme1n1[1] nvme0n1[0]
2097152 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
[==>..................] resync = 11.0% (232704/2097152) finish=0.1min speed=232704K/sec
mdadm -If nvme3n1
mdadm: set nvme3n1 faulty in md127
mdadm: hot removed nvme3n1 from md127
cat /proc/mdstat
Personalities : [raid10]
md127 : active (auto-read-only) raid10 nvme2n1[2] nvme1n1[1] nvme0n1[0]
2097152 blocks super 1.2 512K chunks 2 near-copies [4/3] [UUU_]
resync=PENDING
cat /sys/block/md127/md/resync_start
465408
At the moment it stops the resync. When new disk is added to the array, the
recovery starts and completes, however no resync for the first 2 disks takes
place and array is reported as clean when it's really out-of-sync.
My kernel version is 4.11.
What is the expected behaviour? Shall resync continue on 3-disk RAID10 or
shall it be restarted when recovery completes?
Regards,
Tomek
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox