* Re: [PATCH] IMSM: Correct --examine output for 4k disks
From: Jes Sorensen @ 2017-05-09 16:04 UTC (permalink / raw)
To: Mariusz Dabrowski, linux-raid; +Cc: Maksymilian Kunt
In-Reply-To: <1494331407-2049-1-git-send-email-mariusz.dabrowski@intel.com>
On 05/09/2017 08:03 AM, Mariusz Dabrowski wrote:
> From: Maksymilian Kunt <maksymilian.kunt@intel.com>
>
> "Array Size" and "Per Dev Size" are incorrect for disks with sector size
> different than 512B.
>
> Calculate "Array Size" and "Per Dev Size" based on sector size. Additionally
> print "Sector Size".
>
> Signed-off-by: Maksymilian Kunt <maksymilian.kunt@intel.com>
> Signed-off-by: Mariusz Dabrowski <mariusz.dabrowski@intel.com>
> ---
> super-intel.c | 7 +++++--
> 1 file changed, 5 insertions(+), 2 deletions(-)
I am a little wary of this change as it changes the order of the output.
Does anyone have tools parsing this information which could break from this?
Jes
> diff --git a/super-intel.c b/super-intel.c
> index e13c940..147a70c 100644
> --- a/super-intel.c
> +++ b/super-intel.c
> @@ -1473,13 +1473,16 @@ static void print_imsm_dev(struct intel_super *super,
> ord & IMSM_ORD_REBUILD ? " (out-of-sync)" : "");
> } else
> printf(" This Slot : ?\n");
> + printf(" Sector Size : %u\n", super->sector_size);
> sz = __le32_to_cpu(dev->size_high);
> sz <<= 32;
> sz += __le32_to_cpu(dev->size_low);
> - printf(" Array Size : %llu%s\n", (unsigned long long)sz,
> + printf(" Array Size : %llu%s\n",
> + (unsigned long long)sz * 512 / super->sector_size,
> human_size(sz * 512));
> sz = blocks_per_member(map);
> - printf(" Per Dev Size : %llu%s\n", (unsigned long long)sz,
> + printf(" Per Dev Size : %llu%s\n",
> + (unsigned long long)sz * 512 / super->sector_size,
> human_size(sz * 512));
> printf(" Sector Offset : %llu\n",
> pba_of_lba0(map));
>
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Chris Murphy @ 2017-05-09 16:05 UTC (permalink / raw)
To: Anthony Youngman; +Cc: Nix, Phil Turmel, Ravi (Tom) Hale, Linux-RAID
In-Reply-To: <c5307694-034c-b610-8a27-3bf272cac380@youngman.org.uk>
On Mon, May 8, 2017 at 2:27 PM, Anthony Youngman
<antlists@youngman.org.uk> wrote:
> Yes you have saved a sector sparing. Note that a consumer 3TB drive can
> return, on average, one error every time it's read from end to end 3 times,
> and still be considered "within spec" ie "not faulty" by the manufacturer.
All specs say "less than" which means it's a maximum permissible rate,
not an average. We have no idea what the minimum error rate is - we
being consumers. It's possible high volume users (e.g. Backblaze) have
data on this by now.
> And that's a *brand* *new* drive. That's why building a large array using
> consumer drives is a stupid idea - 4 x 3TB drives and a *within* *spec*
> array must expect to handle at least one error every scrub.
The requirement for any large array is quickly abandoning reattempted
reads in favor of reporting a read error. That's the main reason why
consumer drives are a bad idea, is that it can hang user space waiting
on the long recovery of a drive.
--
Chris Murphy
^ permalink raw reply
* Where to best put the journal device physically?
From: Paul Menzel @ 2017-05-09 16:10 UTC (permalink / raw)
To: linux-raid
Dear Linux RAID folks,
One stupid practical question, for using the journalling support [1].
Currently, enclosures with 16 3.5 inch drives and hardware RAID
controllers are used here. Almost all systems have such a hardware RAID
controller, so moving a enclosure is as simple as plugging the cable
into a different controller card, and updating some maps.
When using a software RAID with an SSD as a journal device, I see the
following options.
1. We need to keep track if a system has a journal device installed,
and if not install one, when moving an enclosure. (Make sure there are
as many slots for SSDs as you have enclosures.)
2. We make sure, in every system, there is a spare SSD drive, usable
as journal device. (Make sure there as many slots for SSDs as you have
enclosures.)
3. There are enclosures, which provide a slot for such a journal
device. Maybe even battery backed?
Are there more options?
It’d be awesome, if you shared, how you deal with this, or point me to
the right direction.
Kind regards,
Paul
[1]
https://kernelnewbies.org/Linux_4.11#head-b75ccfe3ece97a066328c2b2a782434291bf00c6
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Nix @ 2017-05-09 16:11 UTC (permalink / raw)
To: Reindl Harald; +Cc: Phil Turmel, Wols Lists, Ravi (Tom) Hale, linux-raid
In-Reply-To: <5a2d810b-85da-c224-133f-62eab291c161@thelounge.net>
On 9 May 2017, Reindl Harald verbalised:
> Am 09.05.2017 um 13:15 schrieb Nix:
>> (Or were you not running any long self-tests? That's at least as risky
>> as not scrubbing, IMNSHO.)
>
> no i do both regulary
>
> * smart short self-test daily
> * smart long self-test weekly
> * raid scrub weekly
>
> and no - doing a long-smart-test daily is not a good solution, the
> RAID10 array in my office makes *terrible noises* when the SMART
Agreed, though in my case not because of noise, but just because the
test takes fourteen hours and noticeably degrades disk performance while
it runs. I'm doing a long self-test monthly and frankly I'm wondering if
every three months is sufficient.
> well, that machine has not lost a single drive, a clone of it acting
> as homeserver 365/24/7 has lost a dozen in the same time....
A *dozen*?! In six years? Even with a big array you've been incredibly
unlucky, or you have young children and a corresponding disaster rate.
(Meanwhile, my last machine, with much-maligned WD GreenPower
variable-spin-rate disks, was completely happy for eight years, zero
failures, zero reallocations that I can see. I can only hope my new lot
are that good.)
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Reindl Harald @ 2017-05-09 16:46 UTC (permalink / raw)
To: Nix; +Cc: Phil Turmel, Wols Lists, Ravi (Tom) Hale, linux-raid
In-Reply-To: <87ziem3t8a.fsf@esperi.org.uk>
Am 09.05.2017 um 18:11 schrieb Nix:
> On 9 May 2017, Reindl Harald verbalised:
>> and no - doing a long-smart-test daily is not a good solution, the
>> RAID10 array in my office makes *terrible noises* when the SMART
>
> Agreed, though in my case not because of noise, but just because the
> test takes fourteen hours and noticeably degrades disk performance while
> it runs. I'm doing a long self-test monthly and frankly I'm wondering if
> every three months is sufficient.
>
>> well, that machine has not lost a single drive, a clone of it acting
>> as homeserver 365/24/7 has lost a dozen in the same time....
>
> A *dozen*?! In six years? Even with a big array you've been incredibly
> unlucky, or you have young children and a corresponding disaster rate.
> (Meanwhile, my last machine, with much-maligned WD GreenPower
> variable-spin-rate disks, was completely happy for eight years, zero
> failures, zero reallocations that I can see. I can only hope my new lot
> are that good.)
RAID10, 4x2 TB, a rrom temperature of 28 degreee 12 months a year and
214 TB only written - i doubt to be unlucky with that workloads
Filesystem created: Wed Jun 8 13:10:56 2011
Lifetime writes: 214 TB
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Chris Murphy @ 2017-05-09 17:25 UTC (permalink / raw)
To: David Brown
Cc: Nix, Anthony Youngman, Phil Turmel, Ravi (Tom) Hale, Linux-RAID
In-Reply-To: <5911AED4.9030007@hesbynett.no>
On Tue, May 9, 2017 at 5:58 AM, David Brown <david.brown@hesbynett.no> wrote:
> I thought you said that you had read Neil's article. Please go back and
> read it again. If you don't agree with what is written there, then
> there is little more I can say to convince you.
>
> One thing I can try, is to note that you are /not/ the first person to
> think "Surely with RAID-6 we can correct mismatches - it should be
> easy?".
H. Peter Anvin's RAID 6 paper, section 4 is what's apparently under discussion
http://milbret.anydns.info/pub/linux/kernel/people/hpa/raid6.pdf
This is totally non-trivial, especially because it says raid6 cannot
detect or correct more than one corruption, and ensuring that
additional corruption isn't introduced in the rare case is even more
non-trivial.
I do think it's sane for raid6 repair to avoid the current assumption
that data strip is correct, by doing the evaluation in equation 27. If
there's no corruption do nothing, if there's corruption of P or Q then
replace, if there's corruption of data, then report but do not repair
as follows:
1. md reports all data drives and the LBAs for the affected stripe
(otherwise this is not simple if it has to figure out which drive is
actually affected but that's not required, just a matter of better
efficiency in finding out what's really affected.)
2. the file system needs to be able to accept the error from md
3. the file system reports what it negatively impacted: file system
metadata or data and if data, the full filename path.
And now suddenly this work is likewise non-trivial.
And there is already something that will do exactly this: ZFS and
Btrfs. Both can unambiguously, efficiently determine whether data is
corrupt even if a drive doesn't report a read error.
--
Chris Murphy
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Wols Lists @ 2017-05-09 17:49 UTC (permalink / raw)
To: Chris Murphy; +Cc: Nix, Phil Turmel, Ravi (Tom) Hale, Linux-RAID
In-Reply-To: <CAJCQCtQyLjcut=DvPz1QeL8axRMrewkFm3JAOKB66uqWMRHePg@mail.gmail.com>
On 09/05/17 17:05, Chris Murphy wrote:
>> Yes you have saved a sector sparing. Note that a consumer 3TB drive can
>> > return, on average, one error every time it's read from end to end 3 times,
>> > and still be considered "within spec" ie "not faulty" by the manufacturer.
> All specs say "less than" which means it's a maximum permissible rate,
> not an average. We have no idea what the minimum error rate is - we
> being consumers. It's possible high volume users (e.g. Backblaze) have
> data on this by now.
>
In other words, an error rate that high is "acceptable".
And to design software that quite explicitly expects greater perfection
than the hardware itself is guaranteed to provide is, in my humble
opinion, downright negligent!!!
I'm sorry, but like Linus, I take an *engineering* approach to this
stuff, not a mathematical approach. In a mathematical world everything
works perfectly. In an engineering world, things go wrong. You should
always plan for the worst case. But to fail to plan for "the worst
*acceptable* case" is just plain IDIOTIC.
Cheers,
Wol
^ permalink raw reply
* Re: [PATCH 0/5] Don't combine disks with different sector size in the array
From: Jes Sorensen @ 2017-05-09 18:18 UTC (permalink / raw)
To: Tomasz Majchrzak, linux-raid
In-Reply-To: <1494325547-3028-1-git-send-email-tomasz.majchrzak@intel.com>
On 05/09/2017 06:25 AM, Tomasz Majchrzak wrote:
> IMSM doesn't allow to create arrays including drives with different sector
> sizes.
>
> Following patchset blocks all paths to create such configuration:
> * explicit array create
> * array rebuild (spare in the same container, spare moved across containers)
> * array grow
>
> Alexey Obitotskiy (5):
> imsm: allow drives in a container regardless of sector size
> imsm: allocate buffer to support maximum sector size
> imsm: don't allow disks with different sector size in one array
> Allow more spare selection criteria
> Add sector size as spare selection criterion
>
> Incremental.c | 18 ++++++++-----
> Monitor.c | 38 ++++++++++++++++----------
> mdadm.h | 19 +++++++++----
> super-intel.c | 86 ++++++++++++++++++++++++++++++++++++++++-------------------
> util.c | 37 ++++++++++++++++++++++---
> 5 files changed, 140 insertions(+), 58 deletions(-)
>
Applied!
Thanks,
Jes
^ permalink raw reply
* Re: RAID creation resync behaviors
From: Jes Sorensen @ 2017-05-09 18:39 UTC (permalink / raw)
To: Shaohua Li, NeilBrown; +Cc: linux-raid, neilb
In-Reply-To: <20170504020452.kcmjgxnk7zsx7kdx@kernel.org>
On 05/03/2017 10:04 PM, Shaohua Li wrote:
> On Thu, May 04, 2017 at 11:07:01AM +1000, Neil Brown wrote:
>> On Wed, May 03 2017, Shaohua Li wrote:
>>
>>> Hi,
>>>
>>> Currently we have different resync behaviors in array creation.
>>>
>>> - raid1: copy data from disk 0 to disk 1 (overwrite)
>>> - raid10: read both disks, compare and write if there is difference (compare-write)
>>> - raid4/5: read first n-1 disks, calculate parity and then write parity to the last disk (overwrite)
>>> - raid6: read all disks, calculate parity and compare, and write if there is difference (compare-write)
>>
>> The approach taken for raid1 and raid4/5 provides the fastest sync for
>> an array built on uninitialised spinning devices.
>> RAID6 could use the same approach but would involve more CPU and so
>> the original author of the RAID6 code (hpa) chose to go for the low-CPU
>> cost option. I don't know if tests were done, or if they would still be
>> valid on new hardware.
>> The raid10 approach comes from "it is too hard to optimize in general
>> because different RAID10 layouts have different trade-offs, so just
>> take the easy way out."
>
> ok, thanks for the explanation!
>>>
>>> Write whole disk is very unfriendly for SSD, because it reduces lifetime. And
>>> if user already does a trim before creation, the unncessary write could make
>>> SSD slower in the future. Could we prefer compare-write to overwrite if mdadm
>>> detects the disks are SSD? Surely sometimes compare-write is slower than
>>> overwrite, so maybe add new option in mdadm. An option to let mdadm trim SSD
>>> before creation sounds reasonable too.
>>
>> An option to ask mdadm to trim the data space and then --assume-clean
>> certainly sounds reasonable.
>
> This doesn't work well. read returns 0 for trimmed data space in some SSDs, but
> not all. If not, we will have trouble.
/sys/block/<device>/queue/discard_zeroes_data
We could use this as an indicator for what to do.
Jes
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Phil Turmel @ 2017-05-09 19:16 UTC (permalink / raw)
To: Nix, David Brown; +Cc: Anthony Youngman, Ravi (Tom) Hale, linux-raid
In-Reply-To: <878tm65kyx.fsf@esperi.org.uk>
On 05/09/2017 07:27 AM, Nix wrote:
> On 9 May 2017, David Brown uttered the following:
>
>> On 09/05/17 11:53, Nix wrote:
>>> This turns out not to be the case. See this ten-year-old paper:
>>> <https://indico.cern.ch/event/13797/contributions/1362288/attachments/115080/163419/Data_integrity_v3.pdf>.
>>> Five weeks of doing 2GiB writes on 3000 nodes once every two hours
>>> found, they estimated, 50 errors possibly attributable to disk problems
>>> (sector- or page-size regions of corrupted data) on 1/30th of their
>>> nodes. This is *not* rare and it is hard to imagine that 1/30th of disks
>>> used by CERN deserve discarding. It is better to assume that drives
>>> misdirect writes now and then, and to provide a means of recovering from
>>> them that does not take days of panic. RAID-6 gives you that means: md
>>> should use it.
>>
>> RAID-6 does not help here. You have to understand the types of errors
>> that can occur, the reasons for them, the possibilities for detection,
>> the possibilities for recovery, and what the different layers in the
>> system can do about them.
>>
>> RAID (1/5/6) will let you recover from one or more known failed reads,
>> on the assumption that the driver firmware is correct, memories have no
>> errors, buses have no errors, block writes are atomic, write ordering
>> matches the flush commands, block reads are either correct or marked as
>> failed, etc.
>
> I think you're being too pedantic. Many of these things are known not to
> be true on real hardware, and at least one of them cannot possibly be
> true without a journal (atomic block writes). Nonetheless, the md layer
> is quite happy to rebuild after a failed disk even though the write hole
> might have torn garbage into your data, on the grounds that it
> *probably* did not. If your argument was used everywhere, md would never
> have been started because 100% reliability was not guaranteed.
>
> The same, it seems to me, is true of cases in which one drive in a
> RAID-6 reports a few mismatched blocks. It is true that you don't know
> the cause of the mismatches, but you *do* know which bit of the mismatch
> is wrong and what data should be there, subject only to the assumption
> that sufficiently few drives have made simultaneous mistakes that
> redundancy is preserved. And that's the same assumption RAID >0 makes
> all the time anyway!
You are completely ignoring the fact that reconstruction from P,Q is
mathematically correct only if the entire stripe is written together.
Any software or hardware problem that interrupts a complete stripe write
or a short-circuited P,Q update can and therefore often will deliver a
*wrong* assessment of what device is corrupted. In particular, you
can't even tell which devices got new data and which got old data. Even
worse, cable and controller problems have been known to create patterns
of corruption to the way to one or more drives. You desperately need to
know if this happens to your array. It is not only possible, but
*likely* in systems without ECC ram.
The bottom line is that any kernel that implements the auto-correct you
seem to think is a slam dunk will be shunned by any system administrator
who actually cares about their data. Your obtuseness notwithstanding.
All: Please drop me from future CCs on this thread.
Phil
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Wols Lists @ 2017-05-09 19:44 UTC (permalink / raw)
To: Chris Murphy, David Brown; +Cc: Nix, Phil Turmel, Ravi (Tom) Hale, Linux-RAID
In-Reply-To: <CAJCQCtSaLbp4T+j97Ds6xqPmhVwnncGU2UV2_9HEbEJyr6fy+Q@mail.gmail.com>
On 09/05/17 18:25, Chris Murphy wrote:
> On Tue, May 9, 2017 at 5:58 AM, David Brown <david.brown@hesbynett.no> wrote:
>
>> I thought you said that you had read Neil's article. Please go back and
>> read it again. If you don't agree with what is written there, then
>> there is little more I can say to convince you.
>>
>> One thing I can try, is to note that you are /not/ the first person to
>> think "Surely with RAID-6 we can correct mismatches - it should be
>> easy?".
>
> H. Peter Anvin's RAID 6 paper, section 4 is what's apparently under discussion
> http://milbret.anydns.info/pub/linux/kernel/people/hpa/raid6.pdf
>
> This is totally non-trivial, especially because it says raid6 cannot
> detect or correct more than one corruption, and ensuring that
> additional corruption isn't introduced in the rare case is even more
> non-trivial.
And can I point out that that is just one person's opinion? A
well-informed, respected person true, but it's still just opinion. And
imho the argument that says raid should not repair the data applies
equally against fsck - that shouldn't do any repair either! :-)
>
> I do think it's sane for raid6 repair to avoid the current assumption
> that data strip is correct, by doing the evaluation in equation 27. If
> there's no corruption do nothing, if there's corruption of P or Q then
> replace, if there's corruption of data, then report but do not repair
> as follows:
From an ENGINEERING viewpoint, what is the probability that we get a
two-drive error? And if we do, then there's probably something rather
more serious gone wrong?
>
> 1. md reports all data drives and the LBAs for the affected stripe
> (otherwise this is not simple if it has to figure out which drive is
> actually affected but that's not required, just a matter of better
> efficiency in finding out what's really affected.)
md should report the error AND THE DRIVE THAT APPEARS TO BE FAULTY. (Or
maybe we leave that to the below-mentioned mdfsck.)
That way, if it's a bunch of errors on the same drive we know we've got
a problem with the drive. If we've got a bunch of errors on random
drives, we know the problem is probably elsewhere.
>
> 2. the file system needs to be able to accept the error from md
>
> 3. the file system reports what it negatively impacted: file system
> metadata or data and if data, the full filename path.
>
> And now suddenly this work is likewise non-trivial.
Which is why we keep the filesystem out of this. By all means make md
return a list of dud strips, which a filesystem-level utility can then
interpret, but that isn't md's problem.
>
> And there is already something that will do exactly this: ZFS and
> Btrfs. Both can unambiguously, efficiently determine whether data is
> corrupt even if a drive doesn't report a read error.
>
Or we write an mdfsck program. Just like you shouldn't run fsck with
write privileges on a mounted filesystem, you wouldn't run mdfsck with
filesystems in the array mounted.
At the end of the day, md should never corrupt data by default. Which is
what it sounds like is happening at the moment, if it's assuming the
data sectors are correct and the parity is wrong. If one parity appears
correct then by all means rewrite the second ...
But the current setup, where it's currently quite happy to assume a
single-drive error and rewrite it if it's a parity drive, but it won't
assume a single-drive error and and rewrite it if it's a data drive,
just seems totally wrong. Worse, in the latter case, it seems it
actively prevents fixing the problem by updating the parity and
(probably) corrupting the data.
Report the error, give the user the tools to fix it, and LET THEM sort
it out. Just like we do when we run fsck on a filesystem.
(I know I know, patches welcome :-)
Cheers,
Wol
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Nix @ 2017-05-09 20:01 UTC (permalink / raw)
To: Phil Turmel; +Cc: David Brown, Anthony Youngman, Ravi (Tom) Hale, linux-raid
In-Reply-To: <1f354d53-9eeb-7e00-d2c0-f2fa571cf8c8@turmel.org>
On 9 May 2017, Phil Turmel told this:
> On 05/09/2017 07:27 AM, Nix wrote:
>> The same, it seems to me, is true of cases in which one drive in a
>> RAID-6 reports a few mismatched blocks. It is true that you don't know
>> the cause of the mismatches, but you *do* know which bit of the mismatch
>> is wrong and what data should be there, subject only to the assumption
>> that sufficiently few drives have made simultaneous mistakes that
>> redundancy is preserved. And that's the same assumption RAID >0 makes
>> all the time anyway!
>
> You are completely ignoring the fact that reconstruction from P,Q is
> mathematically correct only if the entire stripe is written together.
Ooh, true.
> Any software or hardware problem that interrupts a complete stripe write
> or a short-circuited P,Q update can and therefore often will deliver a
> *wrong* assessment of what device is corrupted. In particular, you
> can't even tell which devices got new data and which got old data. Even
> worse, cable and controller problems have been known to create patterns
> of corruption to the way to one or more drives. You desperately need to
> know if this happens to your array. It is not only possible, but
> *likely* in systems without ECC ram.
Is this still true if the md cache or PPL is in use? The whole point of
these, after all, is to ensure that stripe writes either happen
completely or not at all. (But, again, that'll only guard against things
like power failure interruptions, not bad cabling. However, again, if
you have bad cabling or a bad controller you can expect to have *lots
and lots* of errors -- a small number of errors are much less likely to
be something of this nature. So, again, a threshold like md already
applies elsewhere might seem to be worthwhile. If you are seeing *lots*
of mismatches, clearly correction is unwise -- heck, writing to the
array at all is unwise, and the whole thing might profitably be
remounted ro. I suspect the filesystems will have been remounted ro by
the kernel by this point in any case.)
The point made elsewhere that all your arguments also apply against fsck
still stands. (Why bother with it? If it gave an error, you have a
kernel bug or a bad disk controller, RAM, or cabling, and nothing on
your filesystem can be trusted! just restore from backup!)
Your arguments are absolutely classic "the perfect is the enemy of the
good" arguments, in my view. I can understand falling into that trap on
a RAID list, it's all about paranoia :) but that doesn't mean I agree
with them. I *have* excellent backups, but that doesn't mean I want to
waste hours to days restoring and/or revalidating everything just
because of a persistent mismatch_cnt > 0 which md won't localize for me
or even try to fix because it *might*, uh... no, as far as I can tell
you're worrying that it might in some cases cause corruption of data
that is *already known to be corrupt*. You'll pardon me if this
possibility does not fill me with fear.
> The bottom line is that any kernel that implements the auto-correct you
> seem to think is a slam dunk will be shunned by any system administrator
> who actually cares about their data. Your obtuseness notwithstanding.
Gee, thanks heaps. Next time I want randomly insulting by someone who
doesn't bother to tell me his actual *arguments* in any message before
the one that starts on the insults, I'll come straight to you.
^ permalink raw reply
* In Direct I/O, no matter how big the read/write, the bio's are all 4KB
From: Chris Worley @ 2017-05-09 20:03 UTC (permalink / raw)
To: linuxraid
I realize DIO needs to be 4KB aligned, but why do large I/Os get
chopped down into 4KB bio's?
Doing a blocktrace of the md device, even though the app is writing at
512KB, the MD driver is receiving 4KB bio's.
Why?
Thanks,
Chris
^ permalink raw reply
* Re: In Direct I/O, no matter how big the read/write, the bio's are all 4KB
From: Christoph Hellwig @ 2017-05-09 20:14 UTC (permalink / raw)
To: Chris Worley; +Cc: linuxraid
In-Reply-To: <CANWz5fgNgnZ6R9md1TwD16nZzhAzX-t8BaOsN5-x-FfEE3_h3Q@mail.gmail.com>
On Tue, May 09, 2017 at 10:03:24AM -1000, Chris Worley wrote:
> I realize DIO needs to be 4KB aligned,
Actually many file systems do support sector size aligned direct I/O,
which could be 512 byte aligned. But as that's not your point :)
> but why do large I/Os get
> chopped down into 4KB bio's?
They shouldn't. What kernel version is this on?
> Doing a blocktrace of the md device, even though the app is writing at
> 512KB, the MD driver is receiving 4KB bio's.
The direct I/O code is using bio_add_page to build the bios it submits.
It might be interesting to see which of the checks in it triggers
an early return after the first page.
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Nix @ 2017-05-09 20:18 UTC (permalink / raw)
To: Chris Murphy
Cc: David Brown, Anthony Youngman, Phil Turmel, Ravi (Tom) Hale,
Linux-RAID
In-Reply-To: <CAJCQCtSaLbp4T+j97Ds6xqPmhVwnncGU2UV2_9HEbEJyr6fy+Q@mail.gmail.com>
On 9 May 2017, Chris Murphy verbalised:
> On Tue, May 9, 2017 at 5:58 AM, David Brown <david.brown@hesbynett.no> wrote:
>
>> I thought you said that you had read Neil's article. Please go back and
>> read it again. If you don't agree with what is written there, then
>> there is little more I can say to convince you.
The entire article is predicated on the assumption that when an
inconsistent stripe is found, fixing it is simple because you can just
fail whichever device is inconsistent... but given that the whole
premise of the article is that *you cannot tell which that is*, I don't
see the point in failing anything.
The first comment in the article is someone noting that md doesn't say
which device is failing, what the location of the error is or anything
else a sysadmin might actually find useful for fixing it. "Hey, you have
an error somewhere on some disk on this multi-terabyte array which might
be data corruption and if a disk fails will be data corruption!" is not
too useful :( The fourth comment notes that the "smart" approach, given
RAID-6, has a significantly higher chance of actually fixing the problem
than the simple approach. I'd call that a fairly important comment...
(Neil said: "Similarly a RAID6 with inconsistent P and Q could well not
be able to identify a single block which is "wrong" and even if it could
there is a small possibility that the identified block isn't wrong, but
the other blocks are all inconsistent in such a way as to accidentally
point to it. The probability of this is rather small, but it is
non-zero". As far as I can tell the probability of this is exactly the
same as that of multiple read errors in a single stripe -- possibly far
lower, if you need not only multiple wrong P and Q values but *precisely
mis-chosen* ones. If that wasn't acceptably rare, you wouldn't be using
RAID-6 to begin with.
I've been talking all the time about a stripe which is singly
inconsistent: either all the data blocks are fine and one of P or Q is
fine, or both P and Q and all but one data block is fine, and the
remaining block is inconsistent with all the rest. Obviously if more
blocks are corrupt, you can do nothing but report it. The redundancy
simply isn't there to attempt repair.)
> H. Peter Anvin's RAID 6 paper, section 4 is what's apparently under discussion
> http://milbret.anydns.info/pub/linux/kernel/people/hpa/raid6.pdf
>
> This is totally non-trivial, especially because it says raid6 cannot
> detect or correct more than one corruption, and ensuring that
> additional corruption isn't introduced in the rare case is even more
> non-trivial.
Yeah. Testing this is the bastard problem, really. Fault injection via
dm is the only approach that seems remotely practical to me.
> I do think it's sane for raid6 repair to avoid the current assumption
> that data strip is correct, by doing the evaluation in equation 27. If
> there's no corruption do nothing, if there's corruption of P or Q then
> replace, if there's corruption of data, then report but do not repair
At least indicate *where* the corruption is in the report. (I'd say
"repair, as a non-default option" for people with a different
availability/P(corruption) tradeoff -- since, after all, if you're using
RAID In the first place you value high availability across disk problems
more than most people do, and there is a difference between one bit of
unreported damage that causes a near-certain restore from backup and
either zero or two of them plus a report with an LBA attached so you
know you need to do something...)
> as follows:
>
> 1. md reports all data drives and the LBAs for the affected stripe
> (otherwise this is not simple if it has to figure out which drive is
> actually affected but that's not required, just a matter of better
> efficiency in finding out what's really affected.)
Yep.
> 2. the file system needs to be able to accept the error from md
It would probably need to report this as an -EIO, but I don't know of
any filesystems that can accept asynchronous reports of errors like
this. You'd need reverse mapping to even stand a chance (a non-default
option on xfs, and of course available on btrfs and zfs too). You'd
need self-healing metadata to stand a chance of doing anything about it.
And god knows what a filesystem is meant to do if part of the file data
vanishes. Replace it with \0? ugh. I'd almost rather have the error
go back out to a monitoring daemon and have it send you an email...
> 3. the file system reports what it negatively impacted: file system
> metadata or data and if data, the full filename path.
>
> And now suddenly this work is likewise non-trivial.
Yeah, it's all the layers stacked up to the filesystem that are buggers
to deal with... and now the optional 'just repair it dammit' approach
seems useful again, if just because it doesn't have to deal with all
these extra layers.
> And there is already something that will do exactly this: ZFS and
> Btrfs. Both can unambiguously, efficiently determine whether data is
> corrupt even if a drive doesn't report a read error.
Yeah. Unfortunately both have their own problems: ZFS reimplements the
page cache and adds massive amounts of ineffiicency in the process, and
btrfs is... well... not really baked enough for the sort of high-
availability system that's going to be running RAID, yet. (Alas!)
(Recent xfs can do the same with metadata, but not data.)
--
NULL && (void)
^ permalink raw reply
* Re: RAID creation resync behaviors
From: NeilBrown @ 2017-05-09 20:30 UTC (permalink / raw)
To: Jes Sorensen, Shaohua Li; +Cc: linux-raid, neilb
In-Reply-To: <1fca5ff4-358a-e0cf-d1a4-fc33ecdcbd62@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2390 bytes --]
On Tue, May 09 2017, Jes Sorensen wrote:
> On 05/03/2017 10:04 PM, Shaohua Li wrote:
>> On Thu, May 04, 2017 at 11:07:01AM +1000, Neil Brown wrote:
>>> On Wed, May 03 2017, Shaohua Li wrote:
>>>
>>>> Hi,
>>>>
>>>> Currently we have different resync behaviors in array creation.
>>>>
>>>> - raid1: copy data from disk 0 to disk 1 (overwrite)
>>>> - raid10: read both disks, compare and write if there is difference (compare-write)
>>>> - raid4/5: read first n-1 disks, calculate parity and then write parity to the last disk (overwrite)
>>>> - raid6: read all disks, calculate parity and compare, and write if there is difference (compare-write)
>>>
>>> The approach taken for raid1 and raid4/5 provides the fastest sync for
>>> an array built on uninitialised spinning devices.
>>> RAID6 could use the same approach but would involve more CPU and so
>>> the original author of the RAID6 code (hpa) chose to go for the low-CPU
>>> cost option. I don't know if tests were done, or if they would still be
>>> valid on new hardware.
>>> The raid10 approach comes from "it is too hard to optimize in general
>>> because different RAID10 layouts have different trade-offs, so just
>>> take the easy way out."
>>
>> ok, thanks for the explanation!
>>>>
>>>> Write whole disk is very unfriendly for SSD, because it reduces lifetime. And
>>>> if user already does a trim before creation, the unncessary write could make
>>>> SSD slower in the future. Could we prefer compare-write to overwrite if mdadm
>>>> detects the disks are SSD? Surely sometimes compare-write is slower than
>>>> overwrite, so maybe add new option in mdadm. An option to let mdadm trim SSD
>>>> before creation sounds reasonable too.
>>>
>>> An option to ask mdadm to trim the data space and then --assume-clean
>>> certainly sounds reasonable.
>>
>> This doesn't work well. read returns 0 for trimmed data space in some SSDs, but
>> not all. If not, we will have trouble.
>
> /sys/block/<device>/queue/discard_zeroes_data
>
> We could use this as an indicator for what to do.
>
According to
Documentation/ABI/testing/sysfs-block
Description:
Will always return 0. Don't rely on any specific behavior
for discards, and don't read this file.
See also
Commit: 48920ff2a5a9 ("block: remove the discard_zeroes_data flag")
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: RAID creation resync behaviors
From: Jes Sorensen @ 2017-05-09 20:49 UTC (permalink / raw)
To: NeilBrown, Shaohua Li; +Cc: linux-raid, neilb
In-Reply-To: <87o9v1n56m.fsf@notabene.neil.brown.name>
On 05/09/2017 04:30 PM, NeilBrown wrote:
> On Tue, May 09 2017, Jes Sorensen wrote:
>
>> On 05/03/2017 10:04 PM, Shaohua Li wrote:
>>> On Thu, May 04, 2017 at 11:07:01AM +1000, Neil Brown wrote:
>>>> On Wed, May 03 2017, Shaohua Li wrote:
>>> This doesn't work well. read returns 0 for trimmed data space in some SSDs, but
>>> not all. If not, we will have trouble.
>>
>> /sys/block/<device>/queue/discard_zeroes_data
>>
>> We could use this as an indicator for what to do.
>>
> According to
>
> Documentation/ABI/testing/sysfs-block
>
> Description:
> Will always return 0. Don't rely on any specific behavior
> for discards, and don't read this file.
>
> See also
> Commit: 48920ff2a5a9 ("block: remove the discard_zeroes_data flag")
Crap!
Back to the drawing board :(
Jes
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Wols Lists @ 2017-05-09 20:52 UTC (permalink / raw)
To: Nix, Chris Murphy; +Cc: David Brown, Ravi (Tom) Hale, Linux-RAID
In-Reply-To: <87fugd4wdv.fsf@esperi.org.uk>
On 09/05/17 21:18, Nix wrote:
> (Neil said: "Similarly a RAID6 with inconsistent P and Q could well not
> be able to identify a single block which is "wrong" and even if it could
> there is a small possibility that the identified block isn't wrong, but
> the other blocks are all inconsistent in such a way as to accidentally
> point to it. The probability of this is rather small, but it is
> non-zero". As far as I can tell the probability of this is exactly the
> same as that of multiple read errors in a single stripe -- possibly far
> lower, if you need not only multiple wrong P and Q values but *precisely
> mis-chosen* ones. If that wasn't acceptably rare, you wouldn't be using
> RAID-6 to begin with.
This to me is the crux of the argument.
What is the probability of CORRECTLY identifying a single-disk error?
What is the probability of WRONGLY mistaking a multi-disk error for a
single-disk error?
My gut instinct is that the second scenario is much less likely. So, in
that case, the current setup is that we DELIBERATELY CORRUPT a
recoverable error because of the TINY risk that we might have got it
wrong. Picking probabilities at random, let's say the first probability
is 99 in a hundred, the second is one in a thousand.
On a four-disk raid-6, that means we're throwing away about 500 chances
of recovering the correct data, so that on one occasion we can avoid
corruption. To me that's an insane trade-off.
Neil goes on about "what if a write fails? What if the power goes down?
What if what if?" Those are the wrong questions!!! The correct question
is "can we identify the difference between a single-disk failure and a
multi-disk failure". We don't care what *caused* that failure.
If the power goes down and only the first disk in a stripe is written,
we can correct it back to what it was. If only the last disk failed to
be written, we can correct it back to what it should have been. If at
least two disks are written and at least two disks are not, CAN WE
DETECT THAT? Surely we can - we don't care how many disks are or aren't
written - in that scenario surely all the parities mess up. In which
case we give up and say "corrupt data". Which is no different from at
present other than at present we fix the parity and pretend nothing is
wrong :-(
The problem is that at present we fix the parity and pretend nothing is
wrong when the reality is we *could* have corrected the data, if we
could have been bothered.
So we have to write an mdfsck. Okay. So we have to make sure that no
filesystems on the array are mounted. Okay, that's a bit harder. So we
have to assume that sysadmins are sensible beings who don't screw things
up - okay that's a lot harder :-) But we shouldn't be throwing away LOTS
of data that's easy to recover, because we MIGHT "recover" data that's
wrong.
Yes, yes, I know - code welcome ... :-)
Cheers,
Wol
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Wols Lists @ 2017-05-09 20:57 UTC (permalink / raw)
To: Nix, Phil Turmel; +Cc: linux-raid
In-Reply-To: <87mval4x6k.fsf@esperi.org.uk>
On 09/05/17 21:01, Nix wrote:
> Gee, thanks heaps. Next time I want randomly insulting by someone who
> doesn't bother to tell me his actual *arguments* in any message before
> the one that starts on the insults, I'll come straight to you.
Nix, much as I don't think people are thinking this through rationally
(they live in the perfect world of maths, not the imperfect world of
engineering), I do NOT think insulting Phil on this list is a good idea.
We all say things we shouldn't - I'm a master at it too :-) but sniping
at a well-respected regular isn't wise ...
Can we all tone it down, please ...
Cheers,
Wol
^ permalink raw reply
* Re: RAID creation resync behaviors
From: Martin K. Petersen @ 2017-05-09 21:03 UTC (permalink / raw)
To: Jes Sorensen; +Cc: NeilBrown, Shaohua Li, linux-raid, neilb
In-Reply-To: <76407cb7-437b-142e-e74b-ddd56f3f4ddb@gmail.com>
Jes,
>> According to
>>
>> Documentation/ABI/testing/sysfs-block
>>
>> Description:
>> Will always return 0. Don't rely on any specific behavior
>> for discards, and don't read this file.
>>
>> See also
>> Commit: 48920ff2a5a9 ("block: remove the discard_zeroes_data flag")
>
> Crap!
>
> Back to the drawing board :(
Discard is now a deallocate hint like it was originally intended.
Behavior is non-deterministic and no guarantees are made wrt. block
contents on subsequent reads.
To zero a block range you should be issuing blkdev_issue_zerooout().
This will use the best zeroing approach given the device characteristics
(TRIM/UNMAP if the device provides hard guarantees, or regular WRITE
SAME which also does the right thing on some SSDs). If none of the fancy
zeroing commands work, you'll fall back to writing zeroes manually.
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply
* A sector-of-mismatch warning patch (was Re: Fault tolerance with badblocks)
From: Nix @ 2017-05-09 21:06 UTC (permalink / raw)
To: Chris Murphy
Cc: David Brown, Anthony Youngman, Phil Turmel, Ravi (Tom) Hale,
Linux-RAID
In-Reply-To: <CAJCQCtSaLbp4T+j97Ds6xqPmhVwnncGU2UV2_9HEbEJyr6fy+Q@mail.gmail.com>
On 9 May 2017, Chris Murphy verbalised:
> 1. md reports all data drives and the LBAs for the affected stripe
Enough rambling from me. Here's a hilariously untested patch against
4.11 (as in I haven't even booted with it: my systems are kind of in
flux right now as I migrate to the md-based server that got me all
concerned about this). It compiles! And it's definitely safer than
trying a repair, and makes it possible to recover from a real mismatch
without losing all your hair in the process, or determine that a
mismatch is spurious or irrelevant. And that's enough for me, frankly.
This is a very rare problem, one hopes.
(It's probably not ideal, because the error is just known to be
somewhere in that stripe, not on that sector, which makes determining
the affected data somewhat harder. But at least you can figure out what
filesystem it's on. :) )
8<------------------------------------------------------------->8
From: Nick Alcock <nick.alcock@oracle.com>
Subject: [PATCH] md: report sector of stripes with check mismatches
This makes it possible, with appropriate filesystem support, for a
sysadmin to tell what is affected by the mismatch, and whether
it should be ignored (if it's inside a swap partition, for
instance).
We ratelimit to prevent log flooding: if there are so many
mismatches that ratelimiting is necessary, the individual messages
are relatively unlikely to be important (either the machine is
swapping like crazy or something is very wrong with the disk).
Signed-off-by: Nick Alcock <nick.alcock@oracle.com>
---
drivers/md/raid5.c | 16 ++++++++++++----
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index ed5cd705b985..bcd2e5150e29 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -3959,10 +3959,14 @@ static void handle_parity_checks5(struct r5conf *conf, struct stripe_head *sh,
set_bit(STRIPE_INSYNC, &sh->state);
else {
atomic64_add(STRIPE_SECTORS, &conf->mddev->resync_mismatches);
- if (test_bit(MD_RECOVERY_CHECK, &conf->mddev->recovery))
+ if (test_bit(MD_RECOVERY_CHECK, &conf->mddev->recovery)) {
/* don't try to repair!! */
set_bit(STRIPE_INSYNC, &sh->state);
- else {
+ pr_warn_ratelimited("%s: mismatch around sector "
+ "%llu\n", __func__,
+ (unsigned long long)
+ sh->sector);
+ } else {
sh->check_state = check_state_compute_run;
set_bit(STRIPE_COMPUTE_RUN, &sh->state);
set_bit(STRIPE_OP_COMPUTE_BLK, &s->ops_request);
@@ -4111,10 +4115,14 @@ static void handle_parity_checks6(struct r5conf *conf, struct stripe_head *sh,
}
} else {
atomic64_add(STRIPE_SECTORS, &conf->mddev->resync_mismatches);
- if (test_bit(MD_RECOVERY_CHECK, &conf->mddev->recovery))
+ if (test_bit(MD_RECOVERY_CHECK, &conf->mddev->recovery)) {
/* don't try to repair!! */
set_bit(STRIPE_INSYNC, &sh->state);
- else {
+ pr_warn_ratelimited("%s: mismatch around sector "
+ "%llu\n", __func__,
+ (unsigned long long)
+ sh->sector);
+ } else {
int *target = &sh->ops.target;
sh->ops.target = -1;
--
2.12.2.212.gea238cf35.dirty
^ permalink raw reply related
* Re: RAID creation resync behaviors
From: Jes Sorensen @ 2017-05-09 21:11 UTC (permalink / raw)
To: Martin K. Petersen; +Cc: NeilBrown, Shaohua Li, linux-raid, neilb
In-Reply-To: <yq1k25prbd4.fsf@oracle.com>
On 05/09/2017 05:03 PM, Martin K. Petersen wrote:
>
> Jes,
>
>>> According to
>>>
>>> Documentation/ABI/testing/sysfs-block
>>>
>>> Description:
>>> Will always return 0. Don't rely on any specific behavior
>>> for discards, and don't read this file.
>>>
>>> See also
>>> Commit: 48920ff2a5a9 ("block: remove the discard_zeroes_data flag")
>>
>> Crap!
>>
>> Back to the drawing board :(
>
> Discard is now a deallocate hint like it was originally intended.
> Behavior is non-deterministic and no guarantees are made wrt. block
> contents on subsequent reads.
>
> To zero a block range you should be issuing blkdev_issue_zerooout().
> This will use the best zeroing approach given the device characteristics
> (TRIM/UNMAP if the device provides hard guarantees, or regular WRITE
> SAME which also does the right thing on some SSDs). If none of the fancy
> zeroing commands work, you'll fall back to writing zeroes manually.
Martin,
This is fine within the kernel, however it is not overly useful for
mdadm to determine which strategy to apply when syncing devices.
Jes
^ permalink raw reply
* Re: RAID creation resync behaviors
From: Martin K. Petersen @ 2017-05-09 21:16 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Martin K. Petersen, NeilBrown, Shaohua Li, linux-raid, neilb
In-Reply-To: <08ba08b3-0e8c-c064-6b35-9084545e2c22@gmail.com>
Jes,
> This is fine within the kernel, however it is not overly useful for
> mdadm to determine which strategy to apply when syncing devices.
BLKZEROOUT
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply
* Re: Fault tolerance with badblocks
From: Nix @ 2017-05-09 21:22 UTC (permalink / raw)
To: Wols Lists; +Cc: Phil Turmel, linux-raid
In-Reply-To: <59122D3F.4060906@youngman.org.uk>
On 9 May 2017, Wols Lists told this:
> On 09/05/17 21:01, Nix wrote:
>> Gee, thanks heaps. Next time I want randomly insulting by someone who
>> doesn't bother to tell me his actual *arguments* in any message before
>> the one that starts on the insults, I'll come straight to you.
>
> Nix, much as I don't think people are thinking this through rationally
> (they live in the perfect world of maths, not the imperfect world of
> engineering), I do NOT think insulting Phil on this list is a good idea.
Errr... sure, but I may be ignorant, but I'm not obtuse. Not as far as I
know, anyway. What I am is sleep-deprived. (It takes a special kind of
nervous wreck to be kept awake by a problem like this, that has never
happened in many years of my using md/raid. I think I'll be kept awake
by the possibility of an asteroid strike or a second Carrington Event
tonight.)
> Can we all tone it down, please ...
Sure! I'm generating untested patches now, is that better? (Probably
not. But they do solve this problem enough to reduce the worry quotient
without actually doing the much-more-complex repair side of things.)
--
NULL && (void)
^ permalink raw reply
* Re: RAID creation resync behaviors
From: Jes Sorensen @ 2017-05-09 21:22 UTC (permalink / raw)
To: Martin K. Petersen; +Cc: NeilBrown, Shaohua Li, linux-raid, neilb
In-Reply-To: <yq1bmr1rasn.fsf@oracle.com>
On 05/09/2017 05:16 PM, Martin K. Petersen wrote:
>
> Jes,
>
>> This is fine within the kernel, however it is not overly useful for
>> mdadm to determine which strategy to apply when syncing devices.
>
> BLKZEROOUT
>
Trying to read the code, as this ioctl doesn't seem to be documented
anywhere I can find.... it looks like this ioctl zeroes out a device.
It doesn't help me obtain the information I need to make a decision in
mdadm as whether to overwrite all or compare+write when resyncing a RAID
array.
Jes
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox