* Re: Reshapeing after 4 Days with 0.0% progress
From: iggy @ 2016-06-27 12:20 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <alpine.DEB.2.02.1606271349280.28955@uplift.swm.pp.se>
Thanks for your reply,..
i tried all of recommendations that i found - but nothing pushed me to a
working solution...
it seems ....the fast solution is recreating... :(
next time i'll do a full backup before manage the array
greetings!
^ permalink raw reply
* URE, link resets, user hostile defaults
From: Chris Murphy @ 2016-06-27 16:42 UTC (permalink / raw)
To: linux-raid
Hi,
Drives with SCT ERC not supported or unset, result in potentially long
error recoveries for marginal or bad sectors: upwards of 180 second
recovers are suggested.
The kernel's SCSI command timer default of 30 seconds, i.e.
cat /sys/block/<dev>/device/timeout
conspires to undermine the deep recovery of most drives now on the
market. This by default misconfiguration results in problems list
regulars are very well aware of. It affects all raid configurations,
and even affects the non-RAID single drive use case. And it does so in
a way that doesn't happen on either Windows or macOS. Basically it is
linux kernel induced data loss, the drive very possibly could present
the requested data upon deep recovery being permitted, but the
kernel's command timer is reached before recovery completes, and
obliterates any possibility of recovering that data. By default.
This now seems to affect the majority of use cases. At one time 30
seconds might have been sane for a world with drives that had less
than 30 second recoveries for bad sectors. But that's no longer the
case.
I'm wondering if anyone has floated the idea of changing the kernels
default SCSI command timer? And if so, if there's a thread discussing
where that was rejected upstream? Or if this exposes other liabilities
that merits an alternative work around for what now amounts to a
defect. Maybe it needs to be a udev rule?
Perhaps ideally what we'd like to have is two timers. A timer that
reports back "slowness" for a drive to complete a queued command,
which could be used by e.g. scrubs to preemptively overwrite those
sectors rather than wait for read errors to happen. And then a timer
with a longer value would be the present timer that results in a link
reset once it's reached.
Thanks,
--
Chris Murphy
^ permalink raw reply
* How assemble this array? All devices listed as spares and busy. Huh? Unclear what this means.
From: dave dowless @ 2016-06-27 18:16 UTC (permalink / raw)
To: linux-raid
How to assemble this array? Note, mdadm.conf has no array line.
Originally this was Raid5 with 4disk (sda1, sdb1, sdc1, sdd1)
The current /proc/mdstat list everything as spares:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : inactive sdc1[2](S) sdd1[4](S) sda1[0](S) sdb1[5](S)
23441559552 blocks super 1.2
The last working /proc/mdstat was (yes with failed disk):
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md0 : active raid5 sdb1[5](S) sdd1[4] sdc1[2] sda1[0](F)
17581168128 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/2] [__UU]
I ended up rebooting, but a force assemble returns all devices are
busy. What gives?
mdadm: /dev/sda1 is busy - skipping
mdadm: /dev/sdb1 is busy - skipping
mdadm: /dev/sdc1 is busy - skipping
mdadm: /dev/sdd1 is busy - skipping
Also know I that I had set sdb1 faulty and it came back as a spare and
rebuilt just prior to this problem.
== Drives examined =========
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 13d20dae:40e86bdf:437f8b51:bf2b4dd8
Name : zippy:0 (local to host zippy)
Creation Time : Thu Oct 1 15:37:52 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 11720779776 (5588.90 GiB 6001.04 GB)
Array Size : 17581168128 (16766.71 GiB 18003.12 GB)
Used Dev Size : 11720778752 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 008c8a3f:9e4295b8:48bdc532:c6c2ab4d
Update Time : Thu Jun 9 21:22:12 2016
Checksum : 6c151ac3 - correct
Events : 406
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 13d20dae:40e86bdf:437f8b51:bf2b4dd8
Name : zippy:0 (local to host zippy)
Creation Time : Thu Oct 1 15:37:52 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 11720779776 (5588.90 GiB 6001.04 GB)
Array Size : 17581168128 (16766.71 GiB 18003.12 GB)
Used Dev Size : 11720778752 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4f7deb5f:32333098:c97b360f:578ea16f
Update Time : Fri Jun 10 11:44:26 2016
Checksum : 6a7750d8 - correct
Events : 412
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : ..AA ('A' == active, '.' == missing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 13d20dae:40e86bdf:437f8b51:bf2b4dd8
Name : zippy:0 (local to host zippy)
Creation Time : Thu Oct 1 15:37:52 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 11720779776 (5588.90 GiB 6001.04 GB)
Array Size : 17581168128 (16766.71 GiB 18003.12 GB)
Used Dev Size : 11720778752 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 60905555:5d6adfcc:0200168e:5bec3d80
Update Time : Fri Jun 10 11:44:26 2016
Checksum : 240c7d4f - correct
Events : 412
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : ..AA ('A' == active, '.' == missing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 13d20dae:40e86bdf:437f8b51:bf2b4dd8
Name : zippy:0 (local to host zippy)
Creation Time : Thu Oct 1 15:37:52 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 11720779776 (5588.90 GiB 6001.04 GB)
Array Size : 17581168128 (16766.71 GiB 18003.12 GB)
Used Dev Size : 11720778752 (5588.90 GiB 6001.04 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : d97144c5:35daac52:2ed50f5a:56b2a67d
Update Time : Fri Jun 10 11:44:26 2016
Checksum : e32b69c8 - correct
Events : 412
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : ..AA ('A' == active, '.' == missing)
^ permalink raw reply
* Re: How assemble this array? All devices listed as spares and busy. Huh? Unclear what this means.
From: Mikael Abrahamsson @ 2016-06-27 18:18 UTC (permalink / raw)
To: dave dowless; +Cc: linux-raid
In-Reply-To: <CAM=c-cABEJMShf0m=evmehcfuCM5pBcp5iFOOAi5V1XRa2ekiQ@mail.gmail.com>
On Mon, 27 Jun 2016, dave dowless wrote:
> I ended up rebooting, but a force assemble returns all devices are busy.
> What gives?
That typically indicates that the array is already running. "mdadm --stop
/dev/mdX", and try again.
There should be nothing in "cat /proc/mdstat" when you do assemble.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Re: URE, link resets, user hostile defaults
From: Hannes Reinecke @ 2016-06-28 6:33 UTC (permalink / raw)
To: Chris Murphy, linux-raid
In-Reply-To: <CAJCQCtT1qi1yLjE3Lor3ScVJx7aX4A_S_iM_=VhDtDCD_C9uqw@mail.gmail.com>
On 06/27/2016 06:42 PM, Chris Murphy wrote:
> Hi,
>
> Drives with SCT ERC not supported or unset, result in potentially long
> error recoveries for marginal or bad sectors: upwards of 180 second
> recovers are suggested.
>
> The kernel's SCSI command timer default of 30 seconds, i.e.
>
> cat /sys/block/<dev>/device/timeout
>
> conspires to undermine the deep recovery of most drives now on the
> market. This by default misconfiguration results in problems list
> regulars are very well aware of. It affects all raid configurations,
> and even affects the non-RAID single drive use case. And it does so in
> a way that doesn't happen on either Windows or macOS. Basically it is
> linux kernel induced data loss, the drive very possibly could present
> the requested data upon deep recovery being permitted, but the
> kernel's command timer is reached before recovery completes, and
> obliterates any possibility of recovering that data. By default.
>
> This now seems to affect the majority of use cases. At one time 30
> seconds might have been sane for a world with drives that had less
> than 30 second recoveries for bad sectors. But that's no longer the
> case.
>
'Majority of use cases'.
Hardly. I'm not aware of any issues here.
The problem with SCT ERC (or TLER or whatever the currrent acronym of
the day is called) is that it's a non-standard setting, where every
vendor basically does its own thing.
Plus you can only influence this on higher end-disks; on others you are
at the mercy of the drive firmware, hoping you got the timeout right.
Can you post a message log detailing this problem?
We surely have ways of influencing the timeout, but first we need to
understand what actually is happening.
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
hare@suse.de +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: block: fix blk_queue_split() resource exhaustion
From: Lars Ellenberg @ 2016-06-28 8:24 UTC (permalink / raw)
To: Mike Snitzer
Cc: Ming Lei, linux-block, Roland Kammerer, Jens Axboe, NeilBrown,
Kent Overstreet, Shaohua Li, Alasdair Kergon,
open list:DEVICE-MAPPER (LVM), Ingo Molnar, Peter Zijlstra,
Takashi Iwai, Jiri Kosina, Zheng Liu, Keith Busch,
Martin K. Petersen, Kirill A. Shutemov, Linux Kernel Mailing List,
open list:BCACHE (BLOCK LAYER CACHE),
open list:SOFTWARE RAID (Multiple Disks) SUPPORT
In-Reply-To: <20160624151547.GA13898@redhat.com>
On Fri, Jun 24, 2016 at 11:15:47AM -0400, Mike Snitzer wrote:
> On Fri, Jun 24 2016 at 10:27am -0400,
> Lars Ellenberg <lars.ellenberg@linbit.com> wrote:
>
> > On Fri, Jun 24, 2016 at 07:36:57PM +0800, Ming Lei wrote:
> > > >
> > > > This is not a theoretical problem.
> > > > At least int DRBD, and an unfortunately high IO concurrency wrt. the
> > > > "max-buffers" setting, without this patch we have a reproducible deadlock.
> > >
> > > Is there any log about the deadlock? And is there any lockdep warning
> > > if it is enabled?
> >
> > In DRBD, to avoid potentially very long internal queues as we wait for
> > our replication peer device and local backend, we limit the number of
> > in-flight bios we accept, and block in our ->make_request_fn() if that
> > number exceeds a configured watermark ("max-buffers").
> >
> > Works fine, as long as we could assume that once our make_request_fn()
> > returns, any bios we "recursively" submitted against the local backend
> > would be dispatched. Which used to be the case.
>
> It'd be useful to know whether this patch fixes your issue:
> https://patchwork.kernel.org/patch/7398411/
I would assume so.
because if current is blocked for any reason,
it will dispatch all bios that are still on current->bio_list
to be processed from other contexts.
Which means we will not deadlock, but make progress,
if the unblock of current depends on processing of those bios.
Also see my other mail on the issue,
where I try to better explain the mechanics of "my" deadlock.
Lars
^ permalink raw reply
* Re: [RFC] block: fix blk_queue_split() resource exhaustion
From: Lars Ellenberg @ 2016-06-28 8:45 UTC (permalink / raw)
To: Ming Lei
Cc: linux-block, Roland Kammerer, Jens Axboe, NeilBrown,
Kent Overstreet, Shaohua Li, Alasdair Kergon, Mike Snitzer,
open list:DEVICE-MAPPER (LVM), Ingo Molnar, Peter Zijlstra,
Takashi Iwai, Jiri Kosina, Zheng Liu, Keith Busch,
Martin K. Petersen, Kirill A. Shutemov, Linux Kernel Mailing List,
open list:BCACHE (BLOCK LAYER CACHE),
open list:SOFTWARE RAID (Multiple Disks) SUPPORT
In-Reply-To: <CACVXFVMtMi9WXYQLa3_XZi7W8ZsJe1GnnTLX+7tRMhFAMUSHtA@mail.gmail.com>
On Sat, Jun 25, 2016 at 05:30:29PM +0800, Ming Lei wrote:
> On Fri, Jun 24, 2016 at 10:27 PM, Lars Ellenberg
> <lars.ellenberg@linbit.com> wrote:
> > On Fri, Jun 24, 2016 at 07:36:57PM +0800, Ming Lei wrote:
> >> >
> >> > This is not a theoretical problem.
> >> > At least int DRBD, and an unfortunately high IO concurrency wrt. the
> >> > "max-buffers" setting, without this patch we have a reproducible deadlock.
> >>
> >> Is there any log about the deadlock? And is there any lockdep warning
> >> if it is enabled?
> >
> > In DRBD, to avoid potentially very long internal queues as we wait for
> > our replication peer device and local backend, we limit the number of
> > in-flight bios we accept, and block in our ->make_request_fn() if that
> > number exceeds a configured watermark ("max-buffers").
> >
> > Works fine, as long as we could assume that once our make_request_fn()
> > returns, any bios we "recursively" submitted against the local backend
> > would be dispatched. Which used to be the case.
> >
> > commit 54efd50 block: make generic_make_request handle arbitrarily sized bios
> > introduced blk_queue_split(), which will split any bio that is
> > violating the queue limits into a smaller piece to be processed
> > right away, and queue the "rest" on current->bio_list to be
> > processed by the iteration in generic_make_request() once the
> > current ->make_request_fn() returns.
> >
> > Before that, any user was supposed to go through bio_add_page(),
> > which would call our merge bvec function, and that should already
> > be sufficient to not violate queue limits.
> >
> > Previously, typically the next in line bio to be processed would
> > be the cloned one we passed down to our backend device, which in
> > case of further stacking devices (e.g. logical volume, raid1 or
> > similar) may again push further bios on that list.
> > All of which would simply be processed in turn.
>
> Could you clarify if the issue is triggered in drbd without dm/md involved?
> Or the issue is always triggered with dm/md over drbd?
>
> As Mike mentioned, there is one known issue with dm-snapshot.
The issue can always be triggered, even if only DRBD is involved.
> If your ->make_request_fn() is called several times, each time
> at least you should dispatch one bio returnd from blk_queue_split(),
> so I don't understand why even one bio isn't dispatched out.
I'll try to "visualize" the mechanics of "my" deadlock here.
Just to clarify the effect, assume we had a driver that
for $reasons would limit the number of in-flight IO to
one single bio.
=== before bio_queue_split()
Previously, if something would want to read some range of blocks,
it would allocate a bio, call bio_add_page() a number of times,
and once the bio was "full", call generic_make_request(),
and fill the next bio, then submit that one.
Stacking: "single_in_flight" (q1) -> "sda" (q2)
generic_make_request(bio) current->bio_list in-flight
B_orig_1 NULL 0
q1->make_request_fn(B_orig_1) empty
1
recursive call, queues: B_1_remapped
iterative call:
q2->make_request_fn(B_1_remapped) empty
-> actually dispatched to hardware
return of generic_make_request(B_orig_1).
B_orig_2
q1->make_request_fn(B_orig_1)
1
blocks, waits for in-flight to drop
...
completion of B_orig_1 0
recursive call, queues: B_2_remapped
iterative call:
q2->make_request_fn(B_2_remapped) empty
-> actually dispatched to hardware
return of generic_make_request(B_orig_2).
=== with bio_queue_split()
Now, uppser layers buils one large bio.
generic_make_request(bio) current->bio_list in-flight
B_orig NULL 0
q1->make_request_fn(B_orig) empty
blk_queue_split(B_orig) splits into
B_orig_r1
B_orig_s1
1
recursive call, queues: B_orig_r1, B_s1_remapped
iterative call:
q1->make_request_fn(B_orig_r1) B_s1_remapped
blocks, waits for in-flight to drop
...
which never happens,
because B_s1_remapped has not even been submitted to q2 yet,
let alone dispatched to hardware.
Obviously we don't limit ourselves to just one request, but with larger
incoming bios, with the number of times we split them, with the number
of stacking layers below us, or even layers below us that *also*
call blk_queue_split (or equivalent open coded clone and split)
themselves to split even further, things get worse.
Lars
^ permalink raw reply
* Re: RAID 10 / 2 Devices Layout question
From: Roman Mamedov @ 2016-06-28 9:25 UTC (permalink / raw)
To: Phil Turmel; +Cc: Paul Roland, linux-raid
In-Reply-To: <5770120D.9050809@turmel.org>
[-- Attachment #1: Type: text/plain, Size: 664 bytes --]
On Sun, 26 Jun 2016 13:34:05 -0400
Phil Turmel <philip@turmel.org> wrote:
> On 06/26/2016 01:20 PM, Paul Roland wrote:
>
> > I have two SSDs, and I would like to use md/raid10 (2devices) for
> > performance reasons.
>
> Ok, that's reasonable.
Maybe I'm missing something, but how is a RAID10 of just two devices is
considered reasonable without any questions or explanation? What is the actual
layout that is expected here, and what benefits (or even differences) does it
have compared to RAID0? I thought you need at least 3 devices (and h/w RAID
controllers might even require 4) for RAID10 to start making sense.
--
With respect,
Roman
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply
* Re: RAID 10 / 2 Devices Layout question
From: keld @ 2016-06-28 12:40 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Phil Turmel, Paul Roland, linux-raid
In-Reply-To: <20160628142502.7e69983d@natsu>
On Tue, Jun 28, 2016 at 02:25:02PM +0500, Roman Mamedov wrote:
> On Sun, 26 Jun 2016 13:34:05 -0400
> Phil Turmel <philip@turmel.org> wrote:
>
> > On 06/26/2016 01:20 PM, Paul Roland wrote:
> >
> > > I have two SSDs, and I would like to use md/raid10 (2devices) for
> > > performance reasons.
> >
> > Ok, that's reasonable.
>
> Maybe I'm missing something, but how is a RAID10 of just two devices is
> considered reasonable without any questions or explanation? What is the actual
> layout that is expected here, and what benefits (or even differences) does it
> have compared to RAID0? I thought you need at least 3 devices (and h/w RAID
> controllers might even require 4) for RAID10 to start making sense.
I think your understanding is a common misunderstanding of the Linux raid10.
There is more on this subject on:
https://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10
In short: yes Linux MD RAID10 is a different beast than standard RAID 1+0,
It can work with improved performance with just 2 disks, and even perform
like RAID0 for the Linux MD RAID10 far layout
See more on https://raid.wiki.kernel.org/index.php/Performance
best regards
Keld
^ permalink raw reply
* Re: URE, link resets, user hostile defaults
From: Chris Murphy @ 2016-06-28 17:33 UTC (permalink / raw)
To: Hannes Reinecke; +Cc: Chris Murphy, linux-raid
In-Reply-To: <57721A47.8070203@suse.de>
On Tue, Jun 28, 2016 at 12:33 AM, Hannes Reinecke <hare@suse.de> wrote:
> On 06/27/2016 06:42 PM, Chris Murphy wrote:
>> Hi,
>>
>> Drives with SCT ERC not supported or unset, result in potentially long
>> error recoveries for marginal or bad sectors: upwards of 180 second
>> recovers are suggested.
>>
>> The kernel's SCSI command timer default of 30 seconds, i.e.
>>
>> cat /sys/block/<dev>/device/timeout
>>
>> conspires to undermine the deep recovery of most drives now on the
>> market. This by default misconfiguration results in problems list
>> regulars are very well aware of. It affects all raid configurations,
>> and even affects the non-RAID single drive use case. And it does so in
>> a way that doesn't happen on either Windows or macOS. Basically it is
>> linux kernel induced data loss, the drive very possibly could present
>> the requested data upon deep recovery being permitted, but the
>> kernel's command timer is reached before recovery completes, and
>> obliterates any possibility of recovering that data. By default.
>>
>> This now seems to affect the majority of use cases. At one time 30
>> seconds might have been sane for a world with drives that had less
>> than 30 second recoveries for bad sectors. But that's no longer the
>> case.
>>
> 'Majority of use cases'.
> Hardly. I'm not aware of any issues here.
This list is prolific with this now common misconfiguration. It
manifests on average about weekly, as a message from libata that it's
"hard resetting link". In every single case where the user is
instructed to either set SCT ERC lower than 30 seconds if possible, or
increase the kernel SCSI command timer well above 30 seconds (180 is
often recommended on this list), suddenly the user's problems start to
go away.
Now the md driver gets an explicit read failure from the drive, after
30 seconds, instead of a link reset. And this includes the LBA for the
bad sector, which is apparently what md wants to write the fixup back
to that drive.
However the manifestation of the problem and the nature of this list
self-selects the user reports. Of course people with failed mdadm
based RAID come here. But this problem is also manifesting on Btrfs
for the same reasons. It also manifests, more rarely, with users who
have just a single drive if the drive does "deep recovery" reads on
marginally bad sectors, but the kernel flips out at 30 seconds
preventing that recovery. Of course not every drive model has such
deep recoveries, but by now it's extremely common. I have yet to see a
single consumer hard drive, ever, configured out of the box with SCT
ERC enabled.
> The problem with SCT ERC (or TLER or whatever the currrent acronym of
> the day is called) is that it's a non-standard setting, where every
> vendor basically does its own thing.
> Plus you can only influence this on higher end-disks; on others you are
> at the mercy of the drive firmware, hoping you got the timeout right.
WDC Scorpio Blue laptop drive supports SCT ERC. But it's disabled. Not
a high end drive.
TOSHIBA MQ01ABD100, also an inexpensive laptop drive, supports SCT
ERC, is disabled, not a high end drive.
Samsung 840 EVO, inexpensive SSD, supports SCT ERC, is disabled, not a
high end drive.
That the maximum recovery time is unpublished or difficult to
determine is beside the point. Clearly 30 seconds for the command
timer isn't long enough or this list wouldn't be full of problems
resulting directly from link resets obscuring the actual problem and
fix: either recovering the data, or explicitly failing with a read
error and an LBA so that md (or even Btrfs) can do their job and
overwrite the bad sector thereby causing in-drive remapping by its
firmware.
When this doesn't happen, those bad sectors just accumulate. And it's
a time bomb for data loss waiting to happen.
> Can you post a message log detailing this problem?
http://www.spinics.net/lists/raid/msg50289.html
There are hundreds, maybe thousands, of these on this list alone in
the form of "raid 5 failure help me recover my data!" because what's
happening is the bad sectors accumulate, finally one drive dies, and
of the remaining drives that survive one or more have one or more bad
sectors that were permitted to persist despite scrubs. And that's
because the kernel is f'ing resetting the goddamn link instead of
waiting for the drive to do its job and either recover the data or
explicitly report a read error.
The 30 second default is simply impatient.
Just over the weekend Phil Turmel posted an email with a bunch of back
reading on the subject of timeout mismatches for someone to read. I've
lost track of how many user emails he's replied to, discovering this
common misconfiguration, and get it straightened out and more often
than not helping the user recover data that otherwise would have been
lost *because* of hard link resetting instead of explicit read errors.
http://www.spinics.net/lists/raid/msg52789.html
He isn't the only list regular who helps educate users tirelessly with
this very repetitive work around for a very old misconfiguration that
as far as I can tell only exists on Linux. And it's the default
behavior.
Now we could say that 30 seconds is already too long, and 180 seconds
is just insane. But that's the reality of how a massive pile of
consumer hard drives actually behave. They can do so called "deep
recoveries" that take minutes during which time they appear to hang.
Usually recoveries don't take minutes. But they can take minutes. And
that's where the problem comes in. I don't see why the user should be
the one punished by the kernel, which is in effect what a 30 second
default command timer is doing.
Perhaps there's a better way to do this than change the default
timeout in the kernel? Maybe what we need is an upstream udev rule
that polls SCT ERC for each drive, and if it's
disabled/unsupported/unknown then it sets a much higher command timer
for that block device. And maybe it only does this on USB and SATA.
For anything enterprise or NAS grade, they do report (at least to
smartctl) SCT ERC in deciseconds. The most common value is 70
deciseconds, so a 30 second command timer is OK. Maybe it could even
be lower but that's a separate optimization conversation.
In any case, the current situation is pretty much crap for the user.
And the idea we can educate users on what to buy isn't working, and
the esoteric crap they need to change to avoid the carnage from this
misconfiguration is still mostly unknown even to seasoned sysadmins
and uber storage geeks. They have no idea this is the way things are,
until they have a problem, come on this list, and get schooled. It's a
big reason why so many people have thrown raid 6 at the problem, which
really just papers over the real issue by throwing more redundancy at
it. But this list has in fact seen raid 6 implosions as a result of
this problem where two drives fail, and a 3rd drive has bad sectors
allowed to accumulate because of this misconfiguration and the array
collapses.
> We surely have ways of influencing the timeout, but first we need to
> understand what actually is happening.
I think the list regulars on this list understand what's actually
happening. Users are buying cheap drives that were never designed for,
or even are explicitly excluded from use in raid 5 or raid 6. But the
problem impacts non-RAID users, linear/concat layouts, and RAID 0.
It even impacts Btrfs DUP profile, where there are two copies of
metadata on disk. If one of those fs metadata sectors reads slow
enough, the drive gets reset, the command queue is flushed and now the
fs has to rerequest everything *and* it has no idea, due to lack of a
read error, where to get the mirrored copy of metadata on that drive,
and no idea where to write it back to in order to fix the slow sector
read.
It screws users who merely use ext4, because instead of getting a slow
computer, they get one that starts to face plant with obscure messages
like link resets. The problem isn't the link. The problem is bad
sectors. But they don't see that message because the link reset
happens before the drive reports the read failure.
Where is Phil and Stan to back me up on this?
--
Chris Murphy
^ permalink raw reply
* Re: RAID 10 / 2 Devices Layout question
From: Phil Turmel @ 2016-06-28 18:16 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Paul Roland, linux-raid
In-Reply-To: <20160628142502.7e69983d@natsu>
On 06/28/2016 05:25 AM, Roman Mamedov wrote:
> On Sun, 26 Jun 2016 13:34:05 -0400
> Phil Turmel <philip@turmel.org> wrote:
>
>> On 06/26/2016 01:20 PM, Paul Roland wrote:
>>
>>> I have two SSDs, and I would like to use md/raid10 (2devices) for
>>> performance reasons.
>>
>> Ok, that's reasonable.
>
> Maybe I'm missing something, but how is a RAID10 of just two devices is
> considered reasonable without any questions or explanation? What is the actual
> layout that is expected here, and what benefits (or even differences) does it
> have compared to RAID0? I thought you need at least 3 devices (and h/w RAID
> controllers might even require 4) for RAID10 to start making sense.
Paul made a statement that isn't obviously a problem -- raid10 on SSDs
makes a blistering fast general purpose redundant setup. I use
precisely this setup for my boot and OS devices. I'd do it for my
databases too if I could afford it. (Can't justify it for my
performance needs -- darn.)
If Paul had asked whether or not to use raid10, I'd have had follow up
questions on his application.
^ permalink raw reply
* Re: URE, link resets, user hostile defaults
From: Phil Turmel @ 2016-06-28 18:28 UTC (permalink / raw)
To: Chris Murphy, Hannes Reinecke; +Cc: linux-raid
In-Reply-To: <CAJCQCtQ=DXWg6qK1+9bwQxrzpKUsRvaiaAnatYR-dupg7JOVzQ@mail.gmail.com>
On 06/28/2016 01:33 PM, Chris Murphy wrote:
> Perhaps there's a better way to do this than change the default
> timeout in the kernel? Maybe what we need is an upstream udev rule
> that polls SCT ERC for each drive, and if it's
> disabled/unsupported/unknown then it sets a much higher command timer
> for that block device. And maybe it only does this on USB and SATA.
> For anything enterprise or NAS grade, they do report (at least to
> smartctl) SCT ERC in deciseconds. The most common value is 70
> deciseconds, so a 30 second command timer is OK. Maybe it could even
> be lower but that's a separate optimization conversation.
When Neil retired from maintainership, I mentioned that I would take a
stab at this. You're right, just setting the kernel default timeout to
180 would be a regression. If I recall correctly, there are network
services that would disconnect if storage stacks could delay that long
before replying, whether good or bad.
So a device discovery process that examines the drive's parameter pages
and makes an intelligent decision would be the way to go. But as you
can see, I haven't dug into the ata & scsi layers to figure it out yet.
It won't hurt my feelings if someone beats me to it.
Phil
^ permalink raw reply
* [PATCH 0/2] Support DAX for device-mapper dm-linear devices
From: Toshi Kani @ 2016-06-28 19:37 UTC (permalink / raw)
To: snitzer-H+wXaHxf7aLQT0dZR+AlfA
Cc: linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-raid-u79uwXL29TY76Z2rM5mHXA,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA, agk-H+wXaHxf7aLQT0dZR+AlfA
I noticed that dm-snap reloads DM table of target mapped-device, which
fails for dax-capable device after dax support is added. Ideally,
adding dax support to dm-snap solves the issue, but it cannot be done
easily. This patch-set allows dm-snap to work with dax-capable target
devices when bio-based operation is used.
dax operation is unsupported with dm-snap, such that:
a) After snapshot is taken, mount with dax option to a target device
or a snapshot device fails. They can be mounted without dax.
b) After snapshot is taken to a dax-mounted target device, any writes
to the target device fails (EIO).
b) can be protected by changing lvcreate to fail when snapshot is
requested to a dax-mounted target device.
- Patch 1 solves an error when lvremove is made to a snapshot device.
- Patch 2 solves an error when lvcreate --snapshot is made to a dax-
capable device.
---
Toshi Kani (2):
1/2 dm: update table type check for dax
2/2 dm snap: add fake origin_direct_access
---
drivers/md/dm-ioctl.c | 11 ++++++++++-
drivers/md/dm-snap.c | 8 ++++++++
2 files changed, 18 insertions(+), 1 deletion(-)
^ permalink raw reply
* [PATCH 1/2] dm: update table type check for dax
From: Toshi Kani @ 2016-06-28 19:37 UTC (permalink / raw)
To: snitzer-H+wXaHxf7aLQT0dZR+AlfA
Cc: linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-raid-u79uwXL29TY76Z2rM5mHXA,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA, agk-H+wXaHxf7aLQT0dZR+AlfA
In-Reply-To: <1467142636-21094-1-git-send-email-toshi.kani-ZPxbGqLxI0U@public.gmane.org>
Allow table type DM_TYPE_BIO_BASED to extend with
DM_TYPE_DAX_BIO_BASED since DM_TYPE_DAX_BIO_BASED
supports bio-based requests.
Signed-off-by: Toshi Kani <toshi.kani-ZPxbGqLxI0U@public.gmane.org>
Cc: Mike Snitzer <snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: Alasdair Kergon <agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: Dan Williams <dan.j.williams-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
drivers/md/dm-ioctl.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/drivers/md/dm-ioctl.c b/drivers/md/dm-ioctl.c
index b59e3459..0f32791 100644
--- a/drivers/md/dm-ioctl.c
+++ b/drivers/md/dm-ioctl.c
@@ -1267,6 +1267,15 @@ static int populate_table(struct dm_table *table,
return dm_table_complete(table);
}
+static bool is_valid_type(unsigned cur, unsigned new)
+{
+ if (cur == new ||
+ (cur == DM_TYPE_BIO_BASED && new == DM_TYPE_DAX_BIO_BASED))
+ return true;
+
+ return false;
+}
+
static int table_load(struct dm_ioctl *param, size_t param_size)
{
int r;
@@ -1309,7 +1318,7 @@ static int table_load(struct dm_ioctl *param, size_t param_size)
DMWARN("unable to set up device queue for new table.");
goto err_unlock_md_type;
}
- } else if (dm_get_md_type(md) != dm_table_get_type(t)) {
+ } else if (!is_valid_type(dm_get_md_type(md), dm_table_get_type(t))) {
DMWARN("can't change device type after initial table load.");
r = -EINVAL;
goto err_unlock_md_type;
^ permalink raw reply related
* [PATCH 2/2] dm snap: add fake origin_direct_access
From: Toshi Kani @ 2016-06-28 19:37 UTC (permalink / raw)
To: snitzer-H+wXaHxf7aLQT0dZR+AlfA
Cc: linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-raid-u79uwXL29TY76Z2rM5mHXA,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA, agk-H+wXaHxf7aLQT0dZR+AlfA
In-Reply-To: <1467142636-21094-1-git-send-email-toshi.kani-ZPxbGqLxI0U@public.gmane.org>
dax-capable mapped-device is marked as DM_TYPE_DAX_BIO_BASED,
which supports both dax and bio-based operations. dm-snap
needs to work with dax-capable device when bio-based operation
is used.
Add fake origin_direct_access() to origin device so that its
origin device is also marked as DM_TYPE_DAX_BIO_BASED for
dax-capable device. This allows to extend target's DM table.
dm-snap works normally when bio-based operation is used.
dm-snap does not support dax operation, and mount with dax
option to a target device or snapshot device fails.
Signed-off-by: Toshi Kani <toshi.kani-ZPxbGqLxI0U@public.gmane.org>
Cc: Mike Snitzer <snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: Alasdair Kergon <agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Cc: Dan Williams <dan.j.williams-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
drivers/md/dm-snap.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/drivers/md/dm-snap.c b/drivers/md/dm-snap.c
index 69ab1ff..c472f04 100644
--- a/drivers/md/dm-snap.c
+++ b/drivers/md/dm-snap.c
@@ -2301,6 +2301,13 @@ static int origin_map(struct dm_target *ti, struct bio *bio)
return do_origin(o->dev, bio);
}
+static long origin_direct_access(struct dm_target *ti, sector_t sector,
+ void __pmem **kaddr, pfn_t *pfn, long size)
+{
+ DMWARN("device does not support dax.");
+ return -EIO;
+}
+
/*
* Set the target "max_io_len" field to the minimum of all the snapshots'
* chunk sizes.
@@ -2360,6 +2367,7 @@ static struct target_type origin_target = {
.postsuspend = origin_postsuspend,
.status = origin_status,
.iterate_devices = origin_iterate_devices,
+ .direct_access = origin_direct_access,
};
static struct target_type snapshot_target = {
^ permalink raw reply related
* Re: [PATCH 0/2] fix dm-snap for dax
From: Kani, Toshimitsu @ 2016-06-28 19:43 UTC (permalink / raw)
To: snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Cc: linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org
In-Reply-To: <1467142636-21094-1-git-send-email-toshi.kani-ZPxbGqLxI0U@public.gmane.org>
Sorry, subject line should be "[PATCH 0/2] fix dm-snap for dax"
-Toshi
On Tue, 2016-06-28 at 13:37 -0600, Toshi Kani wrote:
> I noticed that dm-snap reloads DM table of target mapped-device, which
> fails for dax-capable device after dax support is added. Ideally,
> adding dax support to dm-snap solves the issue, but it cannot be done
> easily. This patch-set allows dm-snap to work with dax-capable target
> devices when bio-based operation is used.
>
> dax operation is unsupported with dm-snap, such that:
> a) After snapshot is taken, mount with dax option to a target device
> or a snapshot device fails. They can be mounted without dax.
> b) After snapshot is taken to a dax-mounted target device, any writes
> to the target device fails (EIO).
>
> b) can be protected by changing lvcreate to fail when snapshot is
> requested to a dax-mounted target device.
>
> - Patch 1 solves an error when lvremove is made to a snapshot device.
> - Patch 2 solves an error when lvcreate --snapshot is made to a dax-
> capable device.
>
> ---
> Toshi Kani (2):
> 1/2 dm: update table type check for dax
> 2/2 dm snap: add fake origin_direct_access
>
> ---
> drivers/md/dm-ioctl.c | 11 ++++++++++-
> drivers/md/dm-snap.c | 8 ++++++++
> 2 files changed, 18 insertions(+), 1 deletion(-)
> _______________________________________________
> Linux-nvdimm mailing list
> Linux-nvdimm@lists.01.org
> https://lists.01.org/mailman/listinfo/linux-nvdimm
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply
* Re: [PATCH 1/2] dm: update table type check for dax
From: Mike Snitzer @ 2016-06-28 20:07 UTC (permalink / raw)
To: Toshi Kani
Cc: linux-nvdimm-y27Ovi1pjclAfugRpC6u6w,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-raid-u79uwXL29TY76Z2rM5mHXA,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA, agk-H+wXaHxf7aLQT0dZR+AlfA
In-Reply-To: <1467142636-21094-2-git-send-email-toshi.kani-ZPxbGqLxI0U@public.gmane.org>
On Tue, Jun 28 2016 at 3:37pm -0400,
Toshi Kani <toshi.kani-ZPxbGqLxI0U@public.gmane.org> wrote:
> Allow table type DM_TYPE_BIO_BASED to extend with
> DM_TYPE_DAX_BIO_BASED since DM_TYPE_DAX_BIO_BASED
> supports bio-based requests.
>
> Signed-off-by: Toshi Kani <toshi.kani-ZPxbGqLxI0U@public.gmane.org>
> Cc: Mike Snitzer <snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: Alasdair Kergon <agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: Dan Williams <dan.j.williams-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
> drivers/md/dm-ioctl.c | 11 ++++++++++-
> 1 file changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/md/dm-ioctl.c b/drivers/md/dm-ioctl.c
> index b59e3459..0f32791 100644
> --- a/drivers/md/dm-ioctl.c
> +++ b/drivers/md/dm-ioctl.c
> @@ -1267,6 +1267,15 @@ static int populate_table(struct dm_table *table,
> return dm_table_complete(table);
> }
>
> +static bool is_valid_type(unsigned cur, unsigned new)
> +{
> + if (cur == new ||
> + (cur == DM_TYPE_BIO_BASED && new == DM_TYPE_DAX_BIO_BASED))
> + return true;
> +
> + return false;
> +}
> +
> static int table_load(struct dm_ioctl *param, size_t param_size)
> {
> int r;
> @@ -1309,7 +1318,7 @@ static int table_load(struct dm_ioctl *param, size_t param_size)
> DMWARN("unable to set up device queue for new table.");
> goto err_unlock_md_type;
> }
> - } else if (dm_get_md_type(md) != dm_table_get_type(t)) {
> + } else if (!is_valid_type(dm_get_md_type(md), dm_table_get_type(t))) {
> DMWARN("can't change device type after initial table load.");
> r = -EINVAL;
> goto err_unlock_md_type;
>
You said in the 0th header: "Patch 1 solves an error when lvremove is
made to a snapshot device."
I'm not seeing why this patch 1 fixes anything specific to snapshot
device removal (but I can see why patch 2 makes snapshot creation
"work"). I'll apply your 2nd patch and see if I can see what you mean.
I actually see this error, without either of your 2 proposed patches
applied, when I try to create a snapshot of a DAX capable LV:
# lvcreate -s -n snap -L 100M pmem/lv
device-mapper: reload ioctl on (253:7) failed: Invalid argument
Failed to lock logical volume pmem/lv.
Aborting. Manual intervention required.
Jun 28 15:57:28 rhel-storage-02 kernel: device-mapper: ioctl: can't change device type after initial table load.
^ permalink raw reply
* Re: [PATCH 1/2] dm: update table type check for dax
From: Kani, Toshimitsu @ 2016-06-28 20:23 UTC (permalink / raw)
To: snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Cc: linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <20160628200714.GC8300-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
On Tue, 2016-06-28 at 16:07 -0400, Mike Snitzer wrote:
> On Tue, Jun 28 2016 at 3:37pm -0400,
> Toshi Kani <toshi.kani@hpe.com> wrote:
:
> You said in the 0th header: "Patch 1 solves an error when lvremove is
> made to a snapshot device."
>
> I'm not seeing why this patch 1 fixes anything specific to snapshot
> device removal (but I can see why patch 2 makes snapshot creation
> "work"). I'll apply your 2nd patch and see if I can see what you mean.
>
> I actually see this error, without either of your 2 proposed patches
> applied, when I try to create a snapshot of a DAX capable LV:
>
> # lvcreate -s -n snap -L 100M pmem/lv
> device-mapper: reload ioctl on (253:7) failed: Invalid argument
> Failed to lock logical volume pmem/lv.
> Aborting. Manual intervention required.
> Jun 28 15:57:28 rhel-storage-02 kernel: device-mapper: ioctl: can't change
> device type after initial table load.
Yes, patch 2 fixes this error.
I have not looked into why lvremove does this, but lvremove to a snapshot
device fails to reload DM table of "<dev>-lvsnap" device (which is marked as
DM_TYPE_BIO_BASED) with DM_TYPE_DAX_BIO_BASED. Patch 1 fixes this error. I
think it also generally makes sense to allow this case.
Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply
* Re: [PATCH 2/2] dm snap: add fake origin_direct_access
From: kbuild test robot @ 2016-06-28 20:27 UTC (permalink / raw)
Cc: kbuild-all, snitzer, agk, dan.j.williams, linux-nvdimm, dm-devel,
linux-raid, linux-kernel, Toshi Kani
In-Reply-To: <1467142636-21094-3-git-send-email-toshi.kani@hpe.com>
[-- Attachment #1: Type: text/plain, Size: 1951 bytes --]
Hi,
[auto build test WARNING on dm/for-next]
[also build test WARNING on v4.7-rc5 next-20160628]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Toshi-Kani/Support-DAX-for-device-mapper-dm-linear-devices/20160629-034110
base: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git for-next
config: x86_64-lkp (attached as .config)
compiler: gcc-4.9 (Debian 4.9.3-14) 4.9.3
reproduce:
# save the attached .config to linux build tree
make ARCH=x86_64
All warnings (new ones prefixed by >>):
drivers/md/dm-snap.c:2370:2: error: unknown field 'direct_access' specified in initializer
.direct_access = origin_direct_access,
^
>> drivers/md/dm-snap.c:2370:2: warning: initialization from incompatible pointer type
drivers/md/dm-snap.c:2370:2: warning: (near initialization for 'origin_target.io_hints')
vim +2370 drivers/md/dm-snap.c
2354 struct dm_origin *o = ti->private;
2355
2356 return fn(ti, o->dev, 0, ti->len, data);
2357 }
2358
2359 static struct target_type origin_target = {
2360 .name = "snapshot-origin",
2361 .version = {1, 9, 0},
2362 .module = THIS_MODULE,
2363 .ctr = origin_ctr,
2364 .dtr = origin_dtr,
2365 .map = origin_map,
2366 .resume = origin_resume,
2367 .postsuspend = origin_postsuspend,
2368 .status = origin_status,
2369 .iterate_devices = origin_iterate_devices,
> 2370 .direct_access = origin_direct_access,
2371 };
2372
2373 static struct target_type snapshot_target = {
2374 .name = "snapshot",
2375 .version = {1, 15, 0},
2376 .module = THIS_MODULE,
2377 .ctr = snapshot_ctr,
2378 .dtr = snapshot_dtr,
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
[-- Attachment #2: .config.gz --]
[-- Type: application/octet-stream, Size: 23378 bytes --]
^ permalink raw reply
* Re: [PATCH 1/2] dm: update table type check for dax
From: kbuild test robot @ 2016-06-28 20:33 UTC (permalink / raw)
To: Toshi Kani
Cc: snitzer-H+wXaHxf7aLQT0dZR+AlfA,
linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-raid-u79uwXL29TY76Z2rM5mHXA,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA, kbuild-all-JC7UmRfGjtg,
agk-H+wXaHxf7aLQT0dZR+AlfA
In-Reply-To: <1467142636-21094-2-git-send-email-toshi.kani-ZPxbGqLxI0U@public.gmane.org>
Hi,
[auto build test ERROR on dm/for-next]
[also build test ERROR on v4.7-rc5 next-20160628]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Toshi-Kani/Support-DAX-for-device-mapper-dm-linear-devices/20160629-034110
base: https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git for-next
config: x86_64-randconfig-s1-06290340 (attached as .config)
compiler: gcc-6 (Debian 6.1.1-1) 6.1.1 20160430
reproduce:
# save the attached .config to linux build tree
make ARCH=x86_64
All errors (new ones prefixed by >>):
drivers/md/dm-ioctl.c: In function 'is_valid_type':
>> drivers/md/dm-ioctl.c:1273:42: error: 'DM_TYPE_DAX_BIO_BASED' undeclared (first use in this function)
(cur == DM_TYPE_BIO_BASED && new == DM_TYPE_DAX_BIO_BASED))
^~~~~~~~~~~~~~~~~~~~~
drivers/md/dm-ioctl.c:1273:42: note: each undeclared identifier is reported only once for each function it appears in
vim +/DM_TYPE_DAX_BIO_BASED +1273 drivers/md/dm-ioctl.c
1267 return dm_table_complete(table);
1268 }
1269
1270 static bool is_valid_type(unsigned cur, unsigned new)
1271 {
1272 if (cur == new ||
> 1273 (cur == DM_TYPE_BIO_BASED && new == DM_TYPE_DAX_BIO_BASED))
1274 return true;
1275
1276 return false;
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
^ permalink raw reply
* Re: URE, link resets, user hostile defaults
From: Wols Lists @ 2016-06-28 20:46 UTC (permalink / raw)
To: Phil Turmel, Chris Murphy, Hannes Reinecke; +Cc: linux-raid
In-Reply-To: <5772C1DE.9000403@turmel.org>
On 28/06/16 19:28, Phil Turmel wrote:
> On 06/28/2016 01:33 PM, Chris Murphy wrote:
>
>> > Perhaps there's a better way to do this than change the default
>> > timeout in the kernel? Maybe what we need is an upstream udev rule
>> > that polls SCT ERC for each drive, and if it's
>> > disabled/unsupported/unknown then it sets a much higher command timer
>> > for that block device. And maybe it only does this on USB and SATA.
>> > For anything enterprise or NAS grade, they do report (at least to
>> > smartctl) SCT ERC in deciseconds. The most common value is 70
>> > deciseconds, so a 30 second command timer is OK. Maybe it could even
>> > be lower but that's a separate optimization conversation.
> When Neil retired from maintainership, I mentioned that I would take a
> stab at this. You're right, just setting the kernel default timeout to
> 180 would be a regression. If I recall correctly, there are network
> services that would disconnect if storage stacks could delay that long
> before replying, whether good or bad.
>
> So a device discovery process that examines the drive's parameter pages
> and makes an intelligent decision would be the way to go. But as you
> can see, I haven't dug into the ata & scsi layers to figure it out yet.
> It won't hurt my feelings if someone beats me to it.
Talking off the top of my head :-) would it be possible to spawn a
kernel thread - if it takes longer than an aggressive time-out - that
just waits for far longer then rewrites it if the read finally completes?
In other words, wait for say the 70 deciseconds, then spawn the rewrite
thread, then continue waiting until whatever timeout. The thread could
actually not even time out but just wait for the drive to time out. If
the drive (eventually) responds rather than timing out then the rewrite
would hopefully fix the potential impending URE.
So we'd need two timeouts really. Timeout 1 says "if it takes longer
than this, do a background rewrite when it finally succeeds", and
timeout 2 says "if it takes longer than this, return an error, but let
the rewrite thread continue to wait".
Cheers,
Wol
^ permalink raw reply
* Re: URE, link resets, user hostile defaults
From: Chris Murphy @ 2016-06-28 22:17 UTC (permalink / raw)
To: Wols Lists; +Cc: Phil Turmel, Chris Murphy, Hannes Reinecke, linux-raid
In-Reply-To: <5772E210.4090007@youngman.org.uk>
On Tue, Jun 28, 2016 at 2:46 PM, Wols Lists <antlists@youngman.org.uk> wrote:
> On 28/06/16 19:28, Phil Turmel wrote:
>> On 06/28/2016 01:33 PM, Chris Murphy wrote:
>>
>>> > Perhaps there's a better way to do this than change the default
>>> > timeout in the kernel? Maybe what we need is an upstream udev rule
>>> > that polls SCT ERC for each drive, and if it's
>>> > disabled/unsupported/unknown then it sets a much higher command timer
>>> > for that block device. And maybe it only does this on USB and SATA.
>>> > For anything enterprise or NAS grade, they do report (at least to
>>> > smartctl) SCT ERC in deciseconds. The most common value is 70
>>> > deciseconds, so a 30 second command timer is OK. Maybe it could even
>>> > be lower but that's a separate optimization conversation.
>> When Neil retired from maintainership, I mentioned that I would take a
>> stab at this. You're right, just setting the kernel default timeout to
>> 180 would be a regression. If I recall correctly, there are network
>> services that would disconnect if storage stacks could delay that long
>> before replying, whether good or bad.
>>
>> So a device discovery process that examines the drive's parameter pages
>> and makes an intelligent decision would be the way to go. But as you
>> can see, I haven't dug into the ata & scsi layers to figure it out yet.
>> It won't hurt my feelings if someone beats me to it.
>
> Talking off the top of my head :-) would it be possible to spawn a
> kernel thread - if it takes longer than an aggressive time-out - that
> just waits for far longer then rewrites it if the read finally completes?
>
> In other words, wait for say the 70 deciseconds, then spawn the rewrite
> thread, then continue waiting until whatever timeout. The thread could
> actually not even time out but just wait for the drive to time out. If
> the drive (eventually) responds rather than timing out then the rewrite
> would hopefully fix the potential impending URE.
I do not think the hang comes from the kernel, but from the drive
itself, during these deep recovery reads. I think the whole drive does
a big fat "look at the hand" while it deeply considers, many, many,
many thousands of times, how the F to recover this one goddamn sector.
And until it recovers it (sometimes wrongly), or gives up and submits
a read error, the drive responds to nothing at all, is my
understanding. And hence why the hard resetting link ends up
happening.
If I'm right, threading this in the kernel won't help. It needs to be
threaded in the drive. And I'm also pretty sure that SAS drives have
command queue independence, don't have this problem, and can have
individual commands cancelled, where SATA is S.O.L.
Over on the Btrfs list someone wondered if this hang can just be
reinterpreted as always being the result of bad sectors, the kernel
knows what's pending in the drive command queue, resets the drive, and
pre-emptively reconstructs and overwrites every single LBA for every
command that was stuck in the queue. And I'm like, well that's not
very accurate is it? That's like taking a baseball bat to a tick.
Assuming an unresponsive drive needs a pile of sectors overwritten
might actually piss off that drive, or its controller, and cause other
problems with the storage stack for all we know.
Anyway...
>
> So we'd need two timeouts really. Timeout 1 says "if it takes longer
> than this, do a background rewrite when it finally succeeds", and
> timeout 2 says "if it takes longer than this, return an error, but let
> the rewrite thread continue to wait".
The idea I had was similar, only applying to storage arrays where
there's redundancy. In that case, the first timeout is an
informational message what LBA range is experiencing a read delay. And
that would permit an upper layer to just preemptively overwrite those
slow LBAs.
This is bad though for the single drive use case, or even
linear/concat, and RAID 0 where the data on the slow sector really
must be read or you get EIO or whatever.
But this sort of work around requires lower layers knowing how the
upper layers are organized and I don't know there's a good way to work
that out.
I think we just poll the drive for SCT ERC and based on what comes
back, make a one size fits all decision for that block device. It can
hardly be much worse than now where "hard resetting link" doesn't
really stand out as an oh fuck moment. It just gets lost in other
kernel messages. At least by 180 seconds, there will be truth in
kernel messages that the drive is having read or write errors, even as
depending services are getting mad at all the delays. They're going to
get delayed anyway, just in a different way.
--
Chris Murphy
^ permalink raw reply
* Re: [PATCH 1/2] dm: update table type check for dax
From: Mike Snitzer @ 2016-06-29 0:40 UTC (permalink / raw)
To: Kani, Toshimitsu
Cc: linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <1467145398.3504.439.camel-ZPxbGqLxI0U@public.gmane.org>
On Tue, Jun 28 2016 at 4:23pm -0400,
Kani, Toshimitsu <toshi.kani-ZPxbGqLxI0U@public.gmane.org> wrote:
> On Tue, 2016-06-28 at 16:07 -0400, Mike Snitzer wrote:
> > On Tue, Jun 28 2016 at 3:37pm -0400,
> > Toshi Kani <toshi.kani-ZPxbGqLxI0U@public.gmane.org> wrote:
> :
> > You said in the 0th header: "Patch 1 solves an error when lvremove is
> > made to a snapshot device."
> >
> > I'm not seeing why this patch 1 fixes anything specific to snapshot
> > device removal (but I can see why patch 2 makes snapshot creation
> > "work"). I'll apply your 2nd patch and see if I can see what you mean.
> >
> > I actually see this error, without either of your 2 proposed patches
> > applied, when I try to create a snapshot of a DAX capable LV:
> >
> > # lvcreate -s -n snap -L 100M pmem/lv
> > device-mapper: reload ioctl on (253:7) failed: Invalid argument
> > Failed to lock logical volume pmem/lv.
> > Aborting. Manual intervention required.
> > Jun 28 15:57:28 rhel-storage-02 kernel: device-mapper: ioctl: can't change
> > device type after initial table load.
>
> Yes, patch 2 fixes this error.
>
> I have not looked into why lvremove does this, but lvremove to a snapshot
> device fails to reload DM table of "<dev>-lvsnap" device (which is marked as
> DM_TYPE_BIO_BASED) with DM_TYPE_DAX_BIO_BASED. Patch 1 fixes this error.
It looks like a strange intermediate state that lvm2 uses during
snapshot removal.
Full listing of snapshot related DM tables (before lvremove):
pmem-lv-real: 0 6086656 linear 259:0 2048
pmem-lv: 0 6086656 snapshot-origin 253:5
pmem-snap-cow: 0 204800 linear 259:0 6088704
pmem-snap: 0 6086656 snapshot 253:5 253:6 P 8
When removing this snapshot we're wanting to be left with:
pmem-lv: 0 6086656 linear 259:0 2048
I augmented the DM core error to be more expressive, resulting in:
device-mapper: ioctl: 253:7: can't change device type (from 1 to 4) after initial table load.
1 is DM_TYPE_BIO_BASED and 4 is DM_TYPE_DAX_BIO_BASED -- which makes
sense given the linear target is DM_TYPE_DAX_BIO_BASED.
The previous DM table for 253:7 was:
pmem-snap: 0 6086656 snapshot 253:5 253:6 P 8
The intermediate table that lvm2 is trying to load for 253:7 is:
0 204800 linear 259:0 6088704
(this linear target was previously pmem-snap-cow)
> I think it also generally makes sense to allow this case.
You're probably right but I need to think about it a little bit more.
^ permalink raw reply
* Re: URE, link resets, user hostile defaults
From: Hannes Reinecke @ 2016-06-29 6:01 UTC (permalink / raw)
To: Chris Murphy; +Cc: linux-raid
In-Reply-To: <CAJCQCtQ=DXWg6qK1+9bwQxrzpKUsRvaiaAnatYR-dupg7JOVzQ@mail.gmail.com>
On 06/28/2016 07:33 PM, Chris Murphy wrote:
> On Tue, Jun 28, 2016 at 12:33 AM, Hannes Reinecke <hare@suse.de> wrote:
>> On 06/27/2016 06:42 PM, Chris Murphy wrote:
>>> Hi,
>>>
>>> Drives with SCT ERC not supported or unset, result in potentially long
>>> error recoveries for marginal or bad sectors: upwards of 180 second
>>> recovers are suggested.
>>>
>>> The kernel's SCSI command timer default of 30 seconds, i.e.
>>>
>>> cat /sys/block/<dev>/device/timeout
>>>
>>> conspires to undermine the deep recovery of most drives now on the
>>> market. This by default misconfiguration results in problems list
>>> regulars are very well aware of. It affects all raid configurations,
>>> and even affects the non-RAID single drive use case. And it does so in
>>> a way that doesn't happen on either Windows or macOS. Basically it is
>>> linux kernel induced data loss, the drive very possibly could present
>>> the requested data upon deep recovery being permitted, but the
>>> kernel's command timer is reached before recovery completes, and
>>> obliterates any possibility of recovering that data. By default.
>>>
>>> This now seems to affect the majority of use cases. At one time 30
>>> seconds might have been sane for a world with drives that had less
>>> than 30 second recoveries for bad sectors. But that's no longer the
>>> case.
>>>
>> 'Majority of use cases'.
>> Hardly. I'm not aware of any issues here.
>
> This list is prolific with this now common misconfiguration. It
> manifests on average about weekly, as a message from libata that it's
> "hard resetting link". In every single case where the user is
> instructed to either set SCT ERC lower than 30 seconds if possible, or
> increase the kernel SCSI command timer well above 30 seconds (180 is
> often recommended on this list), suddenly the user's problems start to
> go away.
>
> Now the md driver gets an explicit read failure from the drive, after
> 30 seconds, instead of a link reset. And this includes the LBA for the
> bad sector, which is apparently what md wants to write the fixup back
> to that drive.
>
> However the manifestation of the problem and the nature of this list
> self-selects the user reports. Of course people with failed mdadm
> based RAID come here. But this problem is also manifesting on Btrfs
> for the same reasons. It also manifests, more rarely, with users who
> have just a single drive if the drive does "deep recovery" reads on
> marginally bad sectors, but the kernel flips out at 30 seconds
> preventing that recovery. Of course not every drive model has such
> deep recoveries, but by now it's extremely common. I have yet to see a
> single consumer hard drive, ever, configured out of the box with SCT
> ERC enabled.
>
So we should rather implement SCT ERC support in libata, and set ERC to
the scsi command timeout, no?
Then the user could tweak the scsi command timeout however he likes it
to, and that timeout would be reflected into the ERC setting.
And then we could add an initialisation bit which reads the current ERC
values, increasing the SCSI command timeout as required.
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
hare@suse.de +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [RESEND PATCH 2/3] bcache: update document info
From: Coly Li @ 2016-06-29 10:16 UTC (permalink / raw)
To: Yijing Wang, axboe, Kent Overstreet
Cc: Eric Wheeler, Coly Li, linux-bcache, linux-raid, linux-kernel
In-Reply-To: <1466561534-17595-1-git-send-email-wangyijing@huawei.com>
在 16/6/22 上午10:12, Yijing Wang 写道:
> There is no return in continue_at(), update the documentation.
>
There are 2 modification of this patch. The first one is about a typo,
it is correct.
But I doubt your second modification is proper. The line removed in your
patch is,
> - * continue_at() also, critically, is a macro that returns the
calling function.
> - * There's good reason for this.
> - *
I think this is exactly what original author wants to say. It does not
mean return a value, it means return to the calling function. And the
bellowed lines explains the reason.
> Signed-off-by: Yijing Wang <wangyijing@huawei.com>
> ---
> drivers/md/bcache/closure.c | 2 +-
> drivers/md/bcache/closure.h | 3 ---
> 2 files changed, 1 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/md/bcache/closure.c b/drivers/md/bcache/closure.c
> index 9eaf1d6..864e673 100644
> --- a/drivers/md/bcache/closure.c
> +++ b/drivers/md/bcache/closure.c
> @@ -112,7 +112,7 @@ bool closure_wait(struct closure_waitlist *waitlist, struct closure *cl)
> EXPORT_SYMBOL(closure_wait);
>
> /**
> - * closure_sync - sleep until a closure a closure has nothing left to wait on
> + * closure_sync - sleep until a closure has nothing left to wait on
Yes, this modification is good.
> *
> * Sleeps until the refcount hits 1 - the thread that's running the closure owns
> * the last refcount.
> diff --git a/drivers/md/bcache/closure.h b/drivers/md/bcache/closure.h
> index 782cc2c..f51188d 100644
> --- a/drivers/md/bcache/closure.h
> +++ b/drivers/md/bcache/closure.h
> @@ -31,9 +31,6 @@
> * passing it, as you might expect, the function to run when nothing is pending
> * and the workqueue to run that function out of.
> *
> - * continue_at() also, critically, is a macro that returns the calling function.
> - * There's good reason for this.
> - *
> * To use safely closures asynchronously, they must always have a refcount while
> * they are running owned by the thread that is running them. Otherwise, suppose
> * you submit some bios and wish to have a function run when they all complete:
>
--
Coly Li
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox