* Re: dm stripe: add DAX support
From: Mike Snitzer @ 2016-07-13 2:01 UTC (permalink / raw)
To: Kani, Toshimitsu, axboe-b10kYP2dOMg
Cc: linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <1468362104.8908.43.camel-ZPxbGqLxI0U@public.gmane.org>
On Tue, Jul 12 2016 at 6:22pm -0400,
Kani, Toshimitsu <toshi.kani-ZPxbGqLxI0U@public.gmane.org> wrote:
> On Fri, 2016-06-24 at 14:29 -0400, Mike Snitzer wrote:
> >
> > BTW, if in your testing you could evaluate/quantify any extra overhead
> > from DM that'd be useful to share. It could be there are bottlenecks
> > that need to be fixed, etc.
>
> Here are some results from fio benchmark. The test is single-threaded and is
> bound to one CPU.
>
> DAX LVM IOPS NOTE
> ---------------------------------------
> Y N 790K
> Y Y 754K 5% overhead with LVM
> N N 567K
> N Y 457K 20% overhead with LVM
>
> DAX: Y: mount -o dax,noatime, N: mount -o noatime
> LVM: Y: dm-linear on pmem0 device, N: pmem0 device
> fio: bs=4k, size=2G, direct=1, rw=randread, numjobs=1
>
> Among the 5% overhead with DAX/LVM, the new DM direct_access interfaces
> account for less than 0.5%.
>
> dm_blk_direct_access 0.28%
> linear_direct_access 0.17%
>
> The average latency increases slightly from 0.93us to 0.95us. I think most of
> the overhead comes from the submit_bio() path, which is used only for
> accessing metadata with DAX. I believe this is due to cloning bio for each
> request in DM. There is 12% more L2 miss in total.
>
> Without DAX, 20% overhead is observed with LVM. Average latency increases
> from 1.39us to 1.82us. Without DAX, bio is cloned for both data and metadata.
Thanks for putting this summary together. Unfortunately none of the DM
changes can be queued for 4.8 until Jens takes the 2 block core patches:
https://patchwork.kernel.org/patch/9196021/
https://patchwork.kernel.org/patch/9196019/
Not sure what the hold up and/or issue is with them. But I've asked
twice (and implicilty a 3rd time here). Hopefully they land in time for
4.8.
Mike
^ permalink raw reply
* Re: [dm-devel] [PATCH v2 1/1] block: fix blk_queue_split() resource exhaustion
From: Eric Wheeler @ 2016-07-13 2:18 UTC (permalink / raw)
To: NeilBrown
Cc: Lars Ellenberg, Jens Axboe, linux-block, Martin K. Petersen,
Mike Snitzer, Peter Zijlstra, Jiri Kosina, Ming Lei,
Kirill A. Shutemov, linux-kernel, linux-raid, Takashi Iwai,
linux-bcache, Zheng Liu, Kent Overstreet, Keith Busch, dm-devel,
Shaohua Li, Ingo Molnar, Alasdair Kergon, Roland Kammerer
In-Reply-To: <87lh17efap.fsf@notabene.neil.brown.name>
On Tue, 12 Jul 2016, NeilBrown wrote:
> On Tue, Jul 12 2016, Lars Ellenberg wrote:
> ....
> >
> > Instead, I suggest to distinguish between recursive calls to
> > generic_make_request(), and pushing back the remainder part in
> > blk_queue_split(), by pointing current->bio_lists to a
> > struct recursion_to_iteration_bio_lists {
> > struct bio_list recursion;
> > struct bio_list queue;
> > }
> >
> > By providing each q->make_request_fn() with an empty "recursion"
> > bio_list, then merging any recursively submitted bios to the
> > head of the "queue" list, we can make the recursion-to-iteration
> > logic in generic_make_request() process deepest level bios first,
> > and "sibling" bios of the same level in "natural" order.
> >
> > Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
> > Signed-off-by: Roland Kammerer <roland.kammerer@linbit.com>
>
> Reviewed-by: NeilBrown <neilb@suse.com>
>
> Thanks again for doing this - I think this is a very significant
> improvement and could allow other simplifications.
Thank you Lars for all of this work!
It seems like there have been many 4.3+ blockdev stacking issues and this
will certainly address some of those (maybe all of them?). (I think we
hit this while trying drbd in 4.4 so we dropped back to 4.1 without
issue.) It would be great to hear 4.4.y stable pick this up if
compatible.
Do you believe that this patch would solve any of the proposals by others
since 4.3 related to bio splitting/large bios? I've been collecting a
list, none of which appear have landed yet as of 4.7-rc7 (but correct me
if I'm wrong):
A. [PATCH v2] block: make sure big bio is splitted into at most 256 bvecs
by Ming Lei: https://patchwork.kernel.org/patch/9169483/
B. block: don't make BLK_DEF_MAX_SECTORS too big
by Shaohua Li: http://www.spinics.net/lists/linux-bcache/msg03525.html
C. [1/3] block: flush queued bios when process blocks to avoid deadlock
by Mikulas Patocka: https://patchwork.kernel.org/patch/9204125/
(was https://patchwork.kernel.org/patch/7398411/)
D. dm-crypt: Fix error with too large bios
by Mikulas Patocka: https://patchwork.kernel.org/patch/9138595/
The A,B,D are known to fix large bio issues when stacking dm+bcache
(though the B,D are trivial and probably necessary even with your patch).
Patch C was mentioned earlier in this thread by Mike Snitzer and you
commented briefly that his patch might solve the issue; given that, and in
the interest of minimizing duplicate effort, which of the following best
describes the situation?
1. Your patch could supersede Mikulas's patch; they address the same
issue.
2. Mikulas's patch addresses different issues such and both patches
should be applied.
3. There is overlap between both your patch and Mikulas's such that both
#1,#2 are true and effort to solve this has been duplicated.
If #3, then what might be done to resolve the overlap?
What are the opinions of the authors and can a consensus be reached so we
can see these pushed upstream with the appropriate stable Cc tags and
ultimately fix 4.4.y?
--
Eric Wheeler
^ permalink raw reply
* Re: [PATCH v2 1/1] block: fix blk_queue_split() resource exhaustion
From: Mike Snitzer @ 2016-07-13 2:32 UTC (permalink / raw)
To: Eric Wheeler
Cc: NeilBrown, Lars Ellenberg, Jens Axboe, linux-block,
Martin K. Petersen, Peter Zijlstra, Jiri Kosina, Ming Lei,
Kirill A. Shutemov, linux-kernel, linux-raid, Takashi Iwai,
linux-bcache, Zheng Liu, Kent Overstreet, Keith Busch, dm-devel,
Shaohua Li, Ingo Molnar, Alasdair Kergon, Roland Kammerer
In-Reply-To: <alpine.LRH.2.11.1607121834220.16228@mail.ewheeler.net>
On Tue, Jul 12 2016 at 10:18pm -0400,
Eric Wheeler <bcache@lists.ewheeler.net> wrote:
> On Tue, 12 Jul 2016, NeilBrown wrote:
>
> > On Tue, Jul 12 2016, Lars Ellenberg wrote:
> > ....
> > >
> > > Instead, I suggest to distinguish between recursive calls to
> > > generic_make_request(), and pushing back the remainder part in
> > > blk_queue_split(), by pointing current->bio_lists to a
> > > struct recursion_to_iteration_bio_lists {
> > > struct bio_list recursion;
> > > struct bio_list queue;
> > > }
> > >
> > > By providing each q->make_request_fn() with an empty "recursion"
> > > bio_list, then merging any recursively submitted bios to the
> > > head of the "queue" list, we can make the recursion-to-iteration
> > > logic in generic_make_request() process deepest level bios first,
> > > and "sibling" bios of the same level in "natural" order.
> > >
> > > Signed-off-by: Lars Ellenberg <lars.ellenberg@linbit.com>
> > > Signed-off-by: Roland Kammerer <roland.kammerer@linbit.com>
> >
> > Reviewed-by: NeilBrown <neilb@suse.com>
> >
> > Thanks again for doing this - I think this is a very significant
> > improvement and could allow other simplifications.
>
> Thank you Lars for all of this work!
>
> It seems like there have been many 4.3+ blockdev stacking issues and this
> will certainly address some of those (maybe all of them?). (I think we
> hit this while trying drbd in 4.4 so we dropped back to 4.1 without
> issue.) It would be great to hear 4.4.y stable pick this up if
> compatible.
>
>
> Do you believe that this patch would solve any of the proposals by others
> since 4.3 related to bio splitting/large bios? I've been collecting a
> list, none of which appear have landed yet as of 4.7-rc7 (but correct me
> if I'm wrong):
>
> A. [PATCH v2] block: make sure big bio is splitted into at most 256 bvecs
> by Ming Lei: https://patchwork.kernel.org/patch/9169483/
>
> B. block: don't make BLK_DEF_MAX_SECTORS too big
> by Shaohua Li: http://www.spinics.net/lists/linux-bcache/msg03525.html
>
> C. [1/3] block: flush queued bios when process blocks to avoid deadlock
> by Mikulas Patocka: https://patchwork.kernel.org/patch/9204125/
> (was https://patchwork.kernel.org/patch/7398411/)
>
> D. dm-crypt: Fix error with too large bios
> by Mikulas Patocka: https://patchwork.kernel.org/patch/9138595/
>
> The A,B,D are known to fix large bio issues when stacking dm+bcache
> (though the B,D are trivial and probably necessary even with your patch).
>
> Patch C was mentioned earlier in this thread by Mike Snitzer and you
> commented briefly that his patch might solve the issue; given that, and in
> the interest of minimizing duplicate effort, which of the following best
> describes the situation?
>
> 1. Your patch could supersede Mikulas's patch; they address the same
> issue.
>
> 2. Mikulas's patch addresses different issues such and both patches
> should be applied.
>
> 3. There is overlap between both your patch and Mikulas's such that both
> #1,#2 are true and effort to solve this has been duplicated.
>
>
> If #3, then what might be done to resolve the overlap?
Mikulas confirmed to me that he believes Lars' v2 patch will fix the
dm-snapshot problem, which is being tracked with this BZ:
https://bugzilla.kernel.org/show_bug.cgi?id=119841
We'll see how testing goes (currently underway).
^ permalink raw reply
* Re: kernel checksumming performance vs actual raid device performance
From: Brad Campbell @ 2016-07-13 3:58 UTC (permalink / raw)
To: Matt Garman, Mdadm
In-Reply-To: <CAJvUf-C-Nr8sSnSPL-5jt1NLOAiZjhZ=bjDRUbX_RjphRL+yWA@mail.gmail.com>
On 13/07/16 05:09, Matt Garman wrote:
> We have a system with a 24-disk raid6 array, using 2TB SSDs. We use
> this system in a workload that is 99.9% read-only (a few small
> writes/day, versus countless reads). This system is an NFS server for
> about 50 compute nodes that continually read its data.
> Perhaps naively, I would expect that second-to-last line:
>
> [ 6.499773] raid6: using algorithm avx2x4 gen() (8648 MB/s)
>
> to indicate what kind of throughput I could expect in a degraded
> state, but clearly that is not right---or I have something
> misconfigured.
>
> So in other words, what does that gen() 8648 MB/s metric mean in terms
> of real-world throughput? Is there a way I can "convert" that number
> to expected throughput of a degraded array?
I can't help you with the throughput calculation, but 24 disks would
imply you are striped across 22 disks.
With a reconstruct read, you of course have to read an entire stripe
from all present disks to reconstruct that sector. So I would assume
your machine IO to go through the roof, in addition to whatever
calculations are required to actually do the reconstruction.
What is your chunk size?
^ permalink raw reply
* Re: Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
From: Michal Hocko @ 2016-07-13 11:21 UTC (permalink / raw)
To: Matthias Dahl; +Cc: linux-raid, linux-mm, dm-devel, linux-kernel
In-Reply-To: <459d501038de4d25db6d140ac5ea5f8d@mail.ud19.udmedia.de>
On Tue 12-07-16 16:56:32, Matthias Dahl wrote:
> Hello Michal...
>
> On 2016-07-12 16:07, Michal Hocko wrote:
>
> > /proc/slabinfo could at least point on who is eating that memory.
>
> Thanks. I have made another test (and thus again put the RAID10 out of
> sync for the 100th time, sigh) and made regular snapshots of slabinfo
> which I have attached to this mail.
>
> > Direct IO doesn't get throttled like buffered IO.
>
> Is buffered i/o not used in both cases if I don't explicitly request
> direct i/o?
>
> dd if=/dev/zero /dev/md126p5 bs=512K
> and dd if=/dev/zero /dev/mapper/test-device bs=512K
OK, I misunderstood your question though. You were mentioning the direct
IO earlier so I thought you were referring to it here as well.
> Given that the test-device is dm-crypt on md125p5. Aren't both using
> buffered i/o?
Yes they are.
> > the number of pages under writeback was more or less same throughout
> > the time but there are some local fluctuations when some pages do get
> > completed.
>
> The pages under writeback are those directly destined for the disk, so
> after dm-crypt had done its encryption?
Those are submitted for the IO. dm-crypt will allocate a "shadow" page
for each of them to perform the encryption and only then submit the IO
to the storage underneath see
http://lkml.kernel.org/r/alpine.LRH.2.02.1607121907160.24806@file01.intranet.prod.int.rdu2.redhat.com
> > If not you can enable allocator trace point for a particular object
> > size (or range of sizes) and see who is requesting them.
>
> If that support is baked into the Fedora provided kernel that is. If
> you could give me a few hints or pointers, how to properly do a allocator
> trace point and get some decent data out of it, that would be nice.
You need to have a kernel with CONFIG_TRACEPOINTS and then enable them
via debugfs. You are interested in kmalloc tracepoint and specify a size
as a filter to only see those that are really interesting. I haven't
checked your slabinfo yet - hope to get to it later today.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
From: Michal Hocko @ 2016-07-13 12:18 UTC (permalink / raw)
To: Matthias Dahl; +Cc: linux-raid, linux-mm, dm-devel, linux-kernel
In-Reply-To: <20160713112126.GH28723@dhcp22.suse.cz>
On Wed 13-07-16 13:21:26, Michal Hocko wrote:
> On Tue 12-07-16 16:56:32, Matthias Dahl wrote:
[...]
> > If that support is baked into the Fedora provided kernel that is. If
> > you could give me a few hints or pointers, how to properly do a allocator
> > trace point and get some decent data out of it, that would be nice.
>
> You need to have a kernel with CONFIG_TRACEPOINTS and then enable them
> via debugfs. You are interested in kmalloc tracepoint and specify a size
> as a filter to only see those that are really interesting. I haven't
> checked your slabinfo yet - hope to get to it later today.
The largest contributors seem to be
$ zcat slabinfo.txt.gz | awk '{printf "%s %d\n" , $1, $6*$15}' | head_and_tail.sh 133 | paste-with-diff.sh | sort -n -k3
initial diff [#pages]
radix_tree_node 3444 2592
debug_objects_cache 388 46159
file_lock_ctx 114 138570
buffer_head 5616 238704
kmalloc-256 328 573164
bio-3 24 1118984
So it seems we are accumulating bios and 256B objects. Buffer heads as
well but so much. Having over 4G worth of bios sounds really suspicious.
Note that they pin pages to be written so this might be consuming the
rest of the unaccounted memory! So the main question is why those bios
do not get dispatched or finished.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Fast Loans
From: Financial Service @ 2016-07-13 12:27 UTC (permalink / raw)
To: Recipients
Hae lainaa 3% vastaus tähän viestiin lisätietoja
Apply for a loan at 3% reply to this Email for more Info
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
From: Matthias Dahl @ 2016-07-13 13:18 UTC (permalink / raw)
To: Michal Hocko; +Cc: linux-raid, linux-mm, dm-devel, linux-kernel, Mike Snitzer
In-Reply-To: <20160713121828.GI28723@dhcp22.suse.cz>
Hello Michal,
many thanks for all your time and help on this issue. It is very much
appreciated and I hope we can track this down somehow.
On 2016-07-13 14:18, Michal Hocko wrote:
> So it seems we are accumulating bios and 256B objects. Buffer heads as
> well but so much. Having over 4G worth of bios sounds really
> suspicious.
> Note that they pin pages to be written so this might be consuming the
> rest of the unaccounted memory! So the main question is why those bios
> do not get dispatched or finished.
Ok. It is the Block IOs that do not get completed. I do get it right
that those bio-3 are already the encrypted data that should be written
out but do not for some reason? I tried to figure this out myself but
couldn't find anything -- what does the number "-3" state? It is the
position in some chain or has it a different meaning?
Do you think a trace like you mentioned would help shed some more light
on this? Or would you recommend something else?
I have also cc' Mike Snitzer who commented on this issue before, maybe
he can see some pattern here as well. Pity that Neil Brown is no longer
available as I think this is also somehow related to it being a Intel
Rapid Storage RAID10... since it is the only way I can reproduce it. :(
Thanks,
Matthias
--
Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu
services: custom software [desktop, mobile, web], server administration
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
From: Michal Hocko @ 2016-07-13 13:47 UTC (permalink / raw)
To: Matthias Dahl; +Cc: linux-raid, linux-mm, dm-devel, linux-kernel, Mike Snitzer
In-Reply-To: <74b9325c37948cf2b460bd759cff23dd@mail.ud19.udmedia.de>
On Wed 13-07-16 15:18:11, Matthias Dahl wrote:
> Hello Michal,
>
> many thanks for all your time and help on this issue. It is very much
> appreciated and I hope we can track this down somehow.
>
> On 2016-07-13 14:18, Michal Hocko wrote:
>
> > So it seems we are accumulating bios and 256B objects. Buffer heads as
> > well but so much. Having over 4G worth of bios sounds really suspicious.
> > Note that they pin pages to be written so this might be consuming the
> > rest of the unaccounted memory! So the main question is why those bios
> > do not get dispatched or finished.
>
> Ok. It is the Block IOs that do not get completed. I do get it right
> that those bio-3 are already the encrypted data that should be written
> out but do not for some reason?
Hard to tell. Maybe they are just allocated and waiting for encryption.
But this is just a wild guessing.
> I tried to figure this out myself but
> couldn't find anything -- what does the number "-3" state? It is the
> position in some chain or has it a different meaning?
$ git grep "kmem_cache_create.*bio"
block/bio-integrity.c: bip_slab = kmem_cache_create("bio_integrity_payload",
so there doesn't seem to be any cache like that in the vanilla kernel.
> Do you think a trace like you mentioned would help shed some more light
> on this? Or would you recommend something else?
Dunno. Seeing who is allocating those bios might be helpful but it won't
tell much about what has happened to them after allocation. The tracing
would be more helpful for a mem leak situation which doesn't seem to be
the case here.
This is getting out of my area of expertise so I am not sure I can help
you much more, I am afraid.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: dm stripe: add DAX support
From: Kani, Toshimitsu @ 2016-07-13 15:03 UTC (permalink / raw)
To: axboe-b10kYP2dOMg@public.gmane.org,
snitzer-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
Cc: dm-devel-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org,
linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-raid-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
agk-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
In-Reply-To: <20160713020100.GA5872-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
On Tue, 2016-07-12 at 22:01 -0400, Mike Snitzer wrote:
> On Tue, Jul 12 2016 at 6:22pm -0400,
> Kani, Toshimitsu <toshi.kani@hpe.com> wrote:
> > On Fri, 2016-06-24 at 14:29 -0400, Mike Snitzer wrote:
> > >
:
> Thanks for putting this summary together. Unfortunately none of the DM
> changes can be queued for 4.8 until Jens takes the 2 block core patches:
> https://patchwork.kernel.org/patch/9196021/
> https://patchwork.kernel.org/patch/9196019/
>
> Not sure what the hold up and/or issue is with them. But I've asked
> twice (and implicilty a 3rd time here). Hopefully they land in time for
> 4.8.
Hi Jens,
Can you take the two patches above? These patches add QUEUE_FLAG_DAX and its
sysfs show func. They allow device-mapper to set dax-capability based on its
configuration, and also allow user applications to check dax-capability via
sysfs.
Thanks,
-Toshi
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply
* Re: Help with assembling a stopped array
From: Phil Turmel @ 2016-07-13 15:29 UTC (permalink / raw)
To: Vegard Haugland, linux-raid
In-Reply-To: <CANz1=TGr3_ayX-CxcjsBx6Q0Ggh5PXp1STmqHW1XuMS8GGE7Lg@mail.gmail.com>
Hello Vegard,
On 07/12/2016 09:23 PM, Vegard Haugland wrote:
> Hi all,
>
> I have a raid6 setup (called /dev/md4) with 10 harddrives, ranged from
> sda to sdj.
You'll have to supply more detail. Uncut mdadm -E /dev/sdXY for each
member device in its current state. smartctl -iA -l scterc /dev/sdX for
each member device's drive.
> One of my drives fell out recently, /dev/sde3/ (not entirely sure how
> this happened), but I readded it as /dev/sdk3, and the array started
> rebuilding. This went on for several hours, and it hit 60% completed
> around 8 hours ago.
>
> However, a few hours after that, I lost both /dev/sdg3 and /dev/sdh3 -
> and the array stopped completely. My OS was also running on that
> array, so my entire system freezed forcing me to reboot.
You would not believe how often we encounter reports like yours where
more member devices fail while trying to rebuild/resync/re-add after a
first failure. There's some reading assignments for you at the at end
of this mail that you *must* read and understand or this array will blow
up again.
> I've now booted into busybox and md tells me that it cannot reassemble
> the array as it can only find 7 out of 10 good drives..
>
> I then read the instructions on
> https://raid.wiki.kernel.org/index.php/RAID_Recovery, I started using
> mdadm --examine and found that 8 out of 10 drives has identical event
> counters. The two that deviates from this is /dev/sdg3 and /dev/sdh3.
> I'm not entirely sure if the disk symbolic names are still the same,
> so I further assume that the 8/10 drives that are still good are the
> ones that were running OK before any of this started happening.
>
> As such, I try forcing a reassemble using mdadm -A /dev/md4 --force.
> When I now run mdadm -D /md4, it lists one device as spare (even
> though my array does not use any spares) - and I guess this is the
> main reason the array doesn't start. Can anyone confirm this, or am i
> missing something?
When you used --assemble --force, did you spell out exactly which
devices to use?
> Also, if I try setting the spare as failed by running mdadm -f
> /dev/md4 /dev/sde3, I get "no such device" (as recalled from human
> memory). Removing it works (mdadm -r /dev/md4 /dev/sde3), but every
> time I reassemble, it gets the "spare" tag. I also tried reassemble
> with force, but the same happened..
>
> The initrd on this computer does not have networking unfortunately, so
> I cannot attach any output from any logs.
Boot from a modern rescueCD or USB drive that lets you save the
requested information and output on the console. Include any dmesg
content that involves these devices. When running mdadm, include the -v
option so your console has more info for us.
> If you agree with my summary so far, what should be my next action? Is
> there anything left to try before running mdadm create (with level 6
> and 2 missing drives)?
Absolutely do not use --create!
mdadm --assemble --force is the correct tool, but you *must* resolve the
underlying reason your devices are failing.
Phil
Readings for timeout mismatch issues: (whole threads if possible)
http://marc.info/?l=linux-raid&m=139050322510249&w=2
http://marc.info/?l=linux-raid&m=135863964624202&w=2
http://marc.info/?l=linux-raid&m=135811522817345&w=1
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=132477199207506
http://marc.info/?l=linux-raid&m=133665797115876&w=2
http://marc.info/?l=linux-raid&m=142487508806844&w=3
http://marc.info/?l=linux-raid&m=144535576302583&w=2
^ permalink raw reply
* Re: Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
From: Matthias Dahl @ 2016-07-13 15:32 UTC (permalink / raw)
To: Michal Hocko; +Cc: linux-raid, linux-mm, dm-devel, linux-kernel, Mike Snitzer
In-Reply-To: <20160713134717.GL28723@dhcp22.suse.cz>
Hello...
On 2016-07-13 15:47, Michal Hocko wrote:
> This is getting out of my area of expertise so I am not sure I can help
> you much more, I am afraid.
That's okay. Thank you so much for investing the time.
For what it is worth, I did some further tests and here is what I came
up with:
If I create the plain dm-crypt device with
--perf-submit_from_crypt_cpus,
I can run the tests for as long as I want but the memory problem never
occurs, meaning buffer/cache increase accordingly and thus free memory
decreases but used mem stays pretty constant low. Yet the problem here
is, the system becomes sluggish and throughput is severely impacted.
ksoftirqd is hovering at 100% the whole time.
Somehow my guess is that normally dm-crypt simply takes every request,
encrypts it and queues it internally by itself. And that queue is then
slowly emptied to the underlying device kernel queue. That is why I am
seeing the exploding increase in used memory (rather than in
buffer/cache)
which in the end causes a OOM situation. But that is just my guess. And
IMHO that is not the right thing to do (tm), as can be seen in this
case.
No matter what, I have no clue how to further diagnose this issue. And
given that I already had unsolvable issues with dm-crypt a couple of
months ago with my old machine where the system simply hang itself or
went OOM when the swap was encrypted and just a few kilobytes needed to
be swapped out, I am not so sure anymore I can trust dm-crypt with a
full disk encryption to the point where I feel "safe"... as-in, nothing
bad will happen or the system won't suddenly hang itself due to it. Or
if a bug is introduced, that it will actually be possible to diagnose it
and help fix it or that it will even be eventually fixed. Which is
really
a pity, since I would really have liked to help solve this. With the
swap issue, I did git bisects, tests, narrowed it down to kernel
versions
when said bug was introduced... but in the end, the bug is still present
as far as I know. :(
I will probably look again into ext4 fs encryption. My whole point is
just that in case any of disks go faulty and needs to be replaced or
sent in for warranty, I don't have to worry about mails, personal or
business data still being left on the device (e.g. if it is no longer
accessible or has reallocated sectors or whatever) in a readable form.
Oh well. Pity, really.
Thanks again,
Matthias
--
Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu
services: custom software [desktop, mobile, web], server administration
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
From: Ondrej Kozina @ 2016-07-13 16:24 UTC (permalink / raw)
To: Matthias Dahl, Michal Hocko
Cc: linux-raid, linux-mm, dm-devel, linux-kernel, Mike Snitzer
In-Reply-To: <a6e48e37cce530f286e6669fdfc0b3f8@mail.ud19.udmedia.de>
On 07/13/2016 05:32 PM, Matthias Dahl wrote:
>
> No matter what, I have no clue how to further diagnose this issue. And
> given that I already had unsolvable issues with dm-crypt a couple of
> months ago with my old machine where the system simply hang itself or
> went OOM when the swap was encrypted and just a few kilobytes needed to
> be swapped out, I am not so sure anymore I can trust dm-crypt with a
> full disk encryption to the point where I feel "safe"... as-in, nothing
> bad will happen or the system won't suddenly hang itself due to it. Or
> if a bug is introduced, that it will actually be possible to diagnose it
> and help fix it or that it will even be eventually fixed. Which is
> really
> a pity, since I would really have liked to help solve this. With the
> swap issue, I did git bisects, tests, narrowed it down to kernel
> versions
> when said bug was introduced... but in the end, the bug is still present
> as far as I know. :(
>
One step after another. Mathias, your original report was not forgotten,
it's just not so easy to find the real culprit and fix it without
causing yet another regression. See the
https://marc.info/?l=linux-mm&m=146825178722612&w=2 thread...
Not to mention that on current 4.7-rc7 kernels it behaves yet slightly
differently (yet far from ideally).
Regards O.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Fwd: kernel checksumming performance vs actual raid device performance
From: Doug Dumitru @ 2016-07-13 16:52 UTC (permalink / raw)
To: linux-raid
In-Reply-To: <CAFx4rwQj3_JTNiS0zsQjp_sPXWkrp0ggjg_UiR7oJ8u0X9PQVA@mail.gmail.com>
---------- Forwarded message ----------
From: Doug Dumitru <doug@easyco.com>
Date: Tue, Jul 12, 2016 at 7:10 PM
Subject: Re: kernel checksumming performance vs actual raid device performance
To: Matt Garman <matthew.garman@gmail.com>
Mr. Garman,
If you only lose a single drive in a raid-6 array, then only XOR
parity needs to be re-computed. The "first" parity drive in RAID-6
array is actually a RAID-5 parity drive. The CPU "parity calc"
overhead for re-computing a missing raid-5 drive is very cheap and
should run at > 5GB/sec.
The raid-6 "test" numbers are the performance of calculating the
raid-6 parity "syndrome". The overhead of calculating a missing disk
with raid-6 is higher.
In terms of performance overhead, most people look at long linear
write performance. In this case, raid-6 calc does matter especially
in that the raid "thread" is singular, so the calcs will saturate a
single thread.
I suspect you are seeing something other than the parity math. I have
24 SSDs in an array here and will need to try this.
You might want to try running "perf" on your system while it is
degraded and see where the thread is churning. I would love to see
those results. I would not be surprised to see that the thread is
literally "spinning". If so, then the 100% cpu is probably fixable,
but it won't actually help performance.
In term of single drive missing performance with short reads, you are
mostly at the mercy of short read IOPS. If you array is reading 8K
blocks at 2GB/sec, this is at 250,000 IOPS and you kill off a drive,
it will jump to 500,000 IOPS. Reading from the good drives remains as
single reads, but read from the missing drives require reads from all
of the others (with raid-5, all but one). I am not sure how the
recovery thread issues these recovery read. Hopefully, it blasts them
at the array with abandon (ie, submit all 22 requests concurrently),
but the code might be less aggressive in deference to hard disks.
SSDs love deep queue depths.
Regardless, 500K IOPS as reads is not that easy. A lot of disk HBAs
start to saturate around there.
A couple of "design" points I would consider, if this is a system that
you need to duplicate.
1) Consider a single CPU socket solution, like an E6-1650 v3.
Multi-socked CPU introduce NUMA and a whole slew of "interesting"
system contention issues.
2) Use good HBA that are direct connected to the disks. I like LSI
3008 and the newer 16-port version, although you need to use only 12
ports with 6GBit SATA/SAS to keep from over-running the PCI-e slot
bandwidth.
3) Do everything you can to hammer deep queue depths.
4) Setup IRQ affinity so that the HBAs spread their IRQ requests across cores.
Doug Dumitru
WildFire Storage
On Tue, Jul 12, 2016 at 2:09 PM, Matt Garman <matthew.garman@gmail.com> wrote:
>
> We have a system with a 24-disk raid6 array, using 2TB SSDs. We use
> this system in a workload that is 99.9% read-only (a few small
> writes/day, versus countless reads). This system is an NFS server for
> about 50 compute nodes that continually read its data.
>
> In a non-degraded state, the system works wonderfully: the md0_raid6
> process uses less than 1% CPU, each drive is around 20% utilization
> (via iostat), no swapping is taking place. The outbound throughput
> averages around 2.0 GB/sec, with 2.5 GB/sec peaks.
>
> However, we had a disk fail, and the throughput dropped considerably,
> with the md0_raid6 process pegged at 100% CPU.
>
> I understand that data from the failed disk will need to be
> reconstructed from parity, and this will cause the md0_raid6 process
> to consume considerable CPU.
>
> What I don't understand is how I can determine what kind of actual MD
> device performance (throughput) I can expect in this state?
>
> Dmesg seems to give some hints:
>
> [ 6.386820] xor: automatically using best checksumming function:
> [ 6.396690] avx : 24064.000 MB/sec
> [ 6.414706] raid6: sse2x1 gen() 7636 MB/s
> [ 6.431725] raid6: sse2x2 gen() 3656 MB/s
> [ 6.448742] raid6: sse2x4 gen() 3917 MB/s
> [ 6.465753] raid6: avx2x1 gen() 5425 MB/s
> [ 6.482766] raid6: avx2x2 gen() 7593 MB/s
> [ 6.499773] raid6: avx2x4 gen() 8648 MB/s
> [ 6.499773] raid6: using algorithm avx2x4 gen() (8648 MB/s)
> [ 6.499774] raid6: using avx2x2 recovery algorithm
>
> (CPU is: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz.)
>
> Perhaps naively, I would expect that second-to-last line:
>
> [ 6.499773] raid6: using algorithm avx2x4 gen() (8648 MB/s)
>
> to indicate what kind of throughput I could expect in a degraded
> state, but clearly that is not right---or I have something
> misconfigured.
>
> So in other words, what does that gen() 8648 MB/s metric mean in terms
> of real-world throughput? Is there a way I can "convert" that number
> to expected throughput of a degraded array?
>
>
> Thanks,
> Matt
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Doug Dumitru
EasyCo LLC
--
Doug Dumitru
EasyCo LLC
^ permalink raw reply
* Re: [dm-devel] Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
From: Matthias Dahl @ 2016-07-13 18:24 UTC (permalink / raw)
To: Ondrej Kozina
Cc: Michal Hocko, linux-raid, linux-mm, dm-devel, linux-kernel,
Mike Snitzer
In-Reply-To: <2704a8dd-48c5-82b8-890e-72bf5e1ed1e1@redhat.com>
Hello Ondrej...
On 2016-07-13 18:24, Ondrej Kozina wrote:
> One step after another.
Sorry, it was not meant to be rude or anything... more frustration
since I cannot be of more help and I really would like to jump in
head-first and help fixing it... but lack the necessary insight into
the kernel internals. But who knows, I started reading Robert Love's
book... so, in a good decade or so. ;-))
> https://marc.info/?l=linux-mm&m=146825178722612&w=2
Thanks for that link. I have to read those more closely tomorrow, since
there are some nice insights into dm-crypt there. :)
Still, you have to admit, it is also rather frustrating/scary if such
a crucial subsystem can have bugs over several major versions that do
result in complete hangs (and can thus cause corruption) and are quite
easily triggerable. It does not instill too much confidence that said
subsystem is so intensively used/tested after all. That's at least how
I feel about it...
So long,
Matthias
--
Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu
services: custom software [desktop, mobile, web], server administration
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: Help with assembling a stopped array
From: Vegard Haugland @ 2016-07-13 19:11 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <caf338d7-cc3f-0aca-e928-7619a64d888c@turmel.org>
Hello Phil.
Thanks for taking your time to answer my post. I really appreciate it!
> You'll have to supply more detail. Uncut mdadm -E /dev/sdXY for each
> member device in its current state. smartctl -iA -l scterc /dev/sdX for
> each member device's drive.
Sure. I wrote that post at 03:23 AM local time somewhat panicked, so I
didn't think of booting a live distro to easily get the logs you
require.
Output from $ for x in /dev/sd?3; do mdadm --examine $x >> output_examine; done;
http://paste.debian.net/780968/
Output from $ for x in /dev/sd?3; do echo $x >> output_smartctl;
smartctl -iA -l scterc $x >> output_smartctl; done;
http://paste.debian.net/780967/
Note that since I booted into a live CD, the device names do no longer
match what I wrote in my original post.
> You would not believe how often we encounter reports like yours where
> more member devices fail while trying to rebuild/resync/re-add after a
> first failure. There's some reading assignments for you at the at end
> of this mail that you *must* read and understand or this array will blow
> up again.
Of the links you posted, only the last one appeared relevant to my
problem. If your intention was that I read these to increase my
knowledge of md in general to avoid issues like mine from happening in
the future, I'm happy to oblige. There were a lot of technical terms
and details in those threads that I don't yet fully grasp, so please
bear with me while I take my time to fully digest this. Do you
recommend I read these in the suggested order?
>
> When you used --assemble --force, did you spell out exactly which
> devices to use?
>
Yes. I spelled out all the good drives and omitted the faulty ones.
The result of this was that one of the drives showed up as spare.
Also, the event counter on all drives specified incremented by one
except the spare one. The current event counters are, as shown while
booting the live CD (again, note that the device names differ from my
original post):
/dev/sda3 Events : 32680547
/dev/sdb3 Events : 32680549
/dev/sdc3 Events : 32679348
/dev/sdd3 Events : 32677935
/dev/sde3 Events : 32680549
/dev/sdg3 Events : 32680549
/dev/sdh3 Events : 32680549
/dev/sdi3 Events : 32680549
/dev/sdj3 Events : 32680549
/dev/sdk3 Events : 32680549
The "spare" (which isn't a spare) is /dev/sda3. I've attempted forcing
assembly twice with no prevail. As such, the event counter has
incremented by 2 on the good drives compared to /dev/sda. The faulty
drives are /dev/sdc3 and /dev/sdd3 as shown by the event counter.
> Boot from a modern rescueCD or USB drive that lets you save the
> requested information and output on the console. Include any dmesg
> content that involves these devices. When running mdadm, include the -v
> option so your console has more info for us.
Noted. Will do that from now on.
> Absolutely do not use --create!
Noted!
Again, thanks for helping out.
Vegard
--
Vegard Haugland <vegard@haugland.at>
Mob. +47 976 68 262
Web: www.v3gard.com | foto.haugland.at
If you want to send me encrypted email, my public GPG key is located
at http://v3gard.com/files/gpg.pub
^ permalink raw reply
* Re: Help with assembling a stopped array
From: Phil Turmel @ 2016-07-13 19:57 UTC (permalink / raw)
To: Vegard Haugland; +Cc: linux-raid
In-Reply-To: <CANz1=TEdGxvDo8WRfCwQy4Ymar2Oo7Ct9acBYJcz06XpaFM15w@mail.gmail.com>
Hi Vegard,
On 07/13/2016 03:11 PM, Vegard Haugland wrote:
> Hello Phil.
>
> Thanks for taking your time to answer my post. I really appreciate it!
>
>> You'll have to supply more detail. Uncut mdadm -E /dev/sdXY for each
>> member device in its current state. smartctl -iA -l scterc /dev/sdX for
>> each member device's drive.
>
> Sure. I wrote that post at 03:23 AM local time somewhat panicked, so I
> didn't think of booting a live distro to easily get the logs you
> require.
>
> Output from $ for x in /dev/sd?3; do mdadm --examine $x >> output_examine; done;
>
> http://paste.debian.net/780968/
The device that's popping up as a spare every time really does have the
spare marker (role #10). That's the drive that was partially re-added
when the array crashed again. Just leave it out of your next attempts.
> Output from $ for x in /dev/sd?3; do echo $x >> output_smartctl;
> smartctl -iA -l scterc $x >> output_smartctl; done;
>
> http://paste.debian.net/780967/
The key is "SCT Error Recovery Control command not supported" on most of
your drives. In fact, upgrading model 7200.11 1T drives to your 2T
Barracudas is precisely the set of models that first bit me in the ass.
I mention that somewhere in the threads I pointed at.
The key is that you have desktop drives in a raid array, and they don't
handle read errors in a way friendly to the linux kernel -- they take
too long. The one 7200.11 drive can be told to timeout quicker (7.0
seconds is typical for spinning disks). The kernel will have to be told
to wait extra long for all of the others. 2-3 minutes. Details in the
reading assignments. The precise sequence of events that breaks MD raid
is described in the sixth:
http://marc.info/?l=linux-raid&m=133665797115876&w=2
> Note that since I booted into a live CD, the device names do no longer
> match what I wrote in my original post.
Yes, device names are not guaranteed to remain constant in linux. MD
stores metadata in a superblock that includes array role (position) and
layout details so the name is superfluous once set up. Getting the
roles wrong with changed device names is a key reason --create is so
dangerous.
After you get your array assembled again, I suggest you run the lsdrv[1]
script to document which drive serial numbers correspond to which array
roles. Also consider using the --update=metadata at some point (not
now) to get away from the v0.90 metadata. It is unreliable when used on
partitions that extend to the end of their parent device.
>> You would not believe how often we encounter reports like yours where
>> more member devices fail while trying to rebuild/resync/re-add after a
>> first failure. There's some reading assignments for you at the at end
>> of this mail that you *must* read and understand or this array will blow
>> up again.
>
> Of the links you posted, only the last one appeared relevant to my
> problem. If your intention was that I read these to increase my
> knowledge of md in general to avoid issues like mine from happening in
> the future, I'm happy to oblige. There were a lot of technical terms
> and details in those threads that I don't yet fully grasp, so please
> bear with me while I take my time to fully digest this. Do you
> recommend I read these in the suggested order?
Yes. Your problem is "timeout mismatch", which is use of drives with
extended error recovery times in any linux software raid array. (I
understand hardware raid also struggles with this, but I don't know the
details.)
For now, use at every boot:
for x in /sys/block/*/device/timeout ; do echo 180 > $x ; done
You can stop doing that when you've replaced all of your desktop drives.
> Yes. I spelled out all the good drives and omitted the faulty ones.
This time, omit the drive that shows up as "spare". Use all nine
others. You really want nine, so the redundancy in your array can
reconstruct when it hits the UREs you obviously have. See "Current
Pending Sector" != 0 in your smartctl reports.
After it assembles the nine, issue "mdadm --run /dev/md4" if it didn't
start. Then "echo check >>/sys/block/md4/md/sync_action".
Wait for that to finish. Then add the spare back to the array.
Also, in the future, paste drive reports and console output inline in
your mails, with word wrapping disabled. That puts the details in the
archives for future googlers to find. (This list allows ~ 100k per mail.)
Phil
^ permalink raw reply
* Re: Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage)
From: Tetsuo Handa @ 2016-07-14 11:18 UTC (permalink / raw)
To: Michal Hocko, Matthias Dahl
Cc: linux-raid, linux-mm, dm-devel, linux-kernel, Mike Snitzer
In-Reply-To: <20160713134717.GL28723@dhcp22.suse.cz>
On 2016/07/13 22:47, Michal Hocko wrote:
> On Wed 13-07-16 15:18:11, Matthias Dahl wrote:
>> I tried to figure this out myself but
>> couldn't find anything -- what does the number "-3" state? It is the
>> position in some chain or has it a different meaning?
>
> $ git grep "kmem_cache_create.*bio"
> block/bio-integrity.c: bip_slab = kmem_cache_create("bio_integrity_payload",
>
> so there doesn't seem to be any cache like that in the vanilla kernel.
>
It is
snprintf(bslab->name, sizeof(bslab->name), "bio-%d", entry);
line in bio_find_or_create_slab() in block/bio.c.
I think you can identify who is creating it by printing backtrace at that line.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [PATCH RFC] md/raid1: fix deadlock between freeze_array() and wait_barrier().
From: Alexander Lyakas @ 2016-07-14 13:35 UTC (permalink / raw)
To: NeilBrown; +Cc: 马建朋, linux-raid, Jes Sorensen, Shaohua Li
In-Reply-To: <87d1miec7q.fsf@notabene.neil.brown.name>
Hello Neil, Shaohua,
On Wed, Jul 13, 2016 at 1:14 AM, NeilBrown <neilb@suse.com> wrote:
> On Tue, Jul 12 2016, Alexander Lyakas wrote:
>
>> Hello Neil,
>>
>> Thank you for your response. I read an email about you retiring from
>> MD/mdadm maintenance and delegating mdadm maintenance to Jes Sorensen.
>> But I was wondering who will be responsible for MD maintenance, and
>> was about to send an email asking that.
>
> Yes, I no longer have maintainership responsibilities, though I'm still
> involved to some extent. Jes Sorensen is looking after mdadm and
> Shaohua Li is looking after the kernel driver (as listed in MAINTAINERS).
>
>
>>
>> On Fri, Jul 8, 2016 at 2:41 AM, NeilBrown <neilb@suse.com> wrote:
>>> On Mon, Jun 27 2016, Alexander Lyakas wrote:
>>>
>>>> When we call wait_barrier, we might have some bios waiting
>>>> in current->bio_list, which prevents the array_freeze call to
>>>> complete. Those can only be internal READs, which have already
>>>> passed the wait_barrier call (thus incrementing nr_pending), but
>>>> still were not submitted to the lower level, due to generic_make_request
>>>> logic to avoid recursive calls. In such case, we have a deadlock:
>>>> - array_frozen is already set to 1, so wait_barrier unconditionally waits, so
>>>> - internal READ bios will not be submitted, thus freeze_array will
>>>> never completes
>>>>
>>>> This problem was originally fixed in commit:
>>>> d6b42dc md/raid1,raid10: avoid deadlock during resync/recovery.
>>>>
>>>> But then it was broken in commit:
>>>> b364e3d raid1: Add a field array_frozen to indicate whether raid in
>>>> freeze state.
>>>
>>> Thanks for the great analysis.
>>> I think this primarily a problem in generic_make_request(). It queues
>>> requests in the *wrong* order.
>>>
>>> Please try the patch from
>>> https://lkml.org/lkml/2016/7/7/428
>>>
>>> and see if it helps. If two requests for a raid1 are in the
>>> generic_make_request queue, this patch causes the sub-requests created
>>> by the first to be handled before the second is attempted.
>> I have read this discussion and more or less (probably less than more)
>> understood that the second patch by Lars is supposed to address our
>> issue. However, we cannot easily apply that patch:
>> - The patch is based on structures added by earlier patch "[RFC]
>> block: fix blk_queue_split() resource exhaustion".
>> - Both patches are not in the mainline tree yet.
>> - Both patches are in block core, which requires to recompile the whole kernel.
>> - Not sure if these patches are applicable for our production kernel
>> 3.18 (long term)
>>
>> I am sure you understand that for production with our current kernel
>> 3.18 (long term) we cannot go with these two patches.
>
> This patch takes the basic concept of those two and applies it just to
> raid1 and raid10. I think it should be sufficient. Can you test? The
> patch is against 3.18
>
> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> index 40b35be34f8d..99208aa2c1c8 100644
> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -1229,7 +1229,7 @@ read_again:
> sectors_handled;
> goto read_again;
> } else
> - generic_make_request(read_bio);
> + bio_list_add_head(current->bio_list, read_bio);
> return;
> }
>
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index 32e282f4c83c..c528102b80b6 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -1288,7 +1288,7 @@ read_again:
> sectors_handled;
> goto read_again;
> } else
> - generic_make_request(read_bio);
> + bio_list_add_head(¤t->bio_list, read_bio);
> return;
> }
Unfortunately, this patch doesn't work. It is super elegant, and seems
like it really should work. But the problem is that the "rdev", to
which we want to send the READ bio, might also be a remapping device
(dm-linear, for example). This device will create its own remapped-bio
and will call generic_make_request(), which will stick the bio onto
current->bio_list TAIL:(:(:( So we are back at square one. This patch
would work if *all* the remapping drivers in the stack were doing
bio_list_add_head() instead of generic_make_request() :(:(:(
It seems the real fix should be that generic_make_request() would use
bio_list_add_head(), but as pointed in
http://www.spinics.net/lists/raid/msg52756.html, there are some
concerns about changing the order of remapped bios.
>
>
>>
>> Since this issue is a real deadlock we are hitting in a long-term 3.18
>> kernel, is there any chance for cc-stable fix? Currently we applied
>> the rudimentary fix I posted. It basically switches context for
>> problematic RAID1 READs, and runs them from a different context. With
>> this fix we don't see the deadlock anymore.
>>
>> Also, can you please comment on another concern I expressed:
>> freeze_array() is now not reentrant. Meaning that if two threads call
>> it in parallel (and it could happen for the same MD), the first thread
>> calling unfreeze_array will mess up things for the second thread.
>
> Yes, that is a regression. This should be enough to fix it. Do you
> agree?
>
> Thanks,
> NeilBrown
>
>
> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> index 40b35be34f8d..5ad25c7d7453 100644
> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -984,7 +984,7 @@ static void freeze_array(struct r1conf *conf, int extra)
> * we continue.
> */
> spin_lock_irq(&conf->resync_lock);
> - conf->array_frozen = 1;
> + conf->array_frozen += 1;
> wait_event_lock_irq_cmd(conf->wait_barrier,
> conf->nr_pending == conf->nr_queued+extra,
> conf->resync_lock,
> @@ -995,7 +995,7 @@ static void unfreeze_array(struct r1conf *conf)
> {
> /* reverse the effect of the freeze */
> spin_lock_irq(&conf->resync_lock);
> - conf->array_frozen = 0;
> + conf->array_frozen -= 1;
> wake_up(&conf->wait_barrier);
> spin_unlock_irq(&conf->resync_lock);
> }
I partially agree. The fix that you suggest makes proper accounting of
whether the array is considered frozen or not.
But the problem is that even with this fix, both threads will think
that they "own" the array. And both will do things, like writing data
or so, which might interfere. The proper fix would ensure that only
one thread "owns" the array, and the second thread waits until the
first calls unfreeze_array(), and then the second thread becomes
"owner" of the array. And of course while there are threads that want
to "freeze" the array, other threads that want to do raise_barrier
etc, should wait.
I am sorry for giving two negative responses in one email:)
Thanks,
Alex.
^ permalink raw reply
* Re: [PATCH RFC] md/raid1: fix deadlock between freeze_array() and wait_barrier().
From: NeilBrown @ 2016-07-14 23:18 UTC (permalink / raw)
To: Alexander Lyakas
Cc: 马建朋, linux-raid, Jes Sorensen, Shaohua Li
In-Reply-To: <CAGRgLy47B1QJrHgGP6L1VFpBwF+g8Uk2QED9MJ1oZr0KNv6kQw@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 6477 bytes --]
On Thu, Jul 14 2016, Alexander Lyakas wrote:
>> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
>> index 32e282f4c83c..c528102b80b6 100644
>> --- a/drivers/md/raid10.c
>> +++ b/drivers/md/raid10.c
>> @@ -1288,7 +1288,7 @@ read_again:
>> sectors_handled;
>> goto read_again;
>> } else
>> - generic_make_request(read_bio);
>> + bio_list_add_head(¤t->bio_list, read_bio);
>> return;
>> }
>
> Unfortunately, this patch doesn't work. It is super elegant, and seems
> like it really should work. But the problem is that the "rdev", to
> which we want to send the READ bio, might also be a remapping device
> (dm-linear, for example). This device will create its own remapped-bio
> and will call generic_make_request(), which will stick the bio onto
> current->bio_list TAIL:(:(:( So we are back at square one. This patch
> would work if *all* the remapping drivers in the stack were doing
> bio_list_add_head() instead of generic_make_request() :(:(:(
>
> It seems the real fix should be that generic_make_request() would use
> bio_list_add_head(), but as pointed in
> http://www.spinics.net/lists/raid/msg52756.html, there are some
> concerns about changing the order of remapped bios.
>
While those concerns are valid, they are about hypothetical performance
issues rather than observed deadlock issues. So I wouldn't be too
worried about them.
However I think you said that you didn't want to touch core code at all
(maybe I misunderstood) so that wouldn't help you anyway.
One option would be to punt the request requests to raidXd:
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 40b35be34f8d..f795e27b2124 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -1229,7 +1229,7 @@ read_again:
sectors_handled;
goto read_again;
} else
- generic_make_request(read_bio);
+ reschedule_retry(r1_bio);
return;
}
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 32e282f4c83c..eec38443075b 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -1288,7 +1288,7 @@ read_again:
sectors_handled;
goto read_again;
} else
- generic_make_request(read_bio);
+ reschedule_retry(r10_bio);
return;
}
That might hurt performance, you would need to measure.
The other approach would be to revert the patch that caused the problem.
e.g.
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 40b35be34f8d..062bb86e5fd8 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -884,7 +884,8 @@ static bool need_to_wait_for_sync(struct r1conf *conf, struct bio *bio)
wait = false;
else
wait = true;
- }
+ } else if (conf->barrier)
+ wait = true;
return wait;
}
>
>>
>>
>>>
>>> Since this issue is a real deadlock we are hitting in a long-term 3.18
>>> kernel, is there any chance for cc-stable fix? Currently we applied
>>> the rudimentary fix I posted. It basically switches context for
>>> problematic RAID1 READs, and runs them from a different context. With
>>> this fix we don't see the deadlock anymore.
>>>
>>> Also, can you please comment on another concern I expressed:
>>> freeze_array() is now not reentrant. Meaning that if two threads call
>>> it in parallel (and it could happen for the same MD), the first thread
>>> calling unfreeze_array will mess up things for the second thread.
>>
>> Yes, that is a regression. This should be enough to fix it. Do you
>> agree?
>>
>> Thanks,
>> NeilBrown
>>
>>
>> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
>> index 40b35be34f8d..5ad25c7d7453 100644
>> --- a/drivers/md/raid1.c
>> +++ b/drivers/md/raid1.c
>> @@ -984,7 +984,7 @@ static void freeze_array(struct r1conf *conf, int extra)
>> * we continue.
>> */
>> spin_lock_irq(&conf->resync_lock);
>> - conf->array_frozen = 1;
>> + conf->array_frozen += 1;
>> wait_event_lock_irq_cmd(conf->wait_barrier,
>> conf->nr_pending == conf->nr_queued+extra,
>> conf->resync_lock,
>> @@ -995,7 +995,7 @@ static void unfreeze_array(struct r1conf *conf)
>> {
>> /* reverse the effect of the freeze */
>> spin_lock_irq(&conf->resync_lock);
>> - conf->array_frozen = 0;
>> + conf->array_frozen -= 1;
>> wake_up(&conf->wait_barrier);
>> spin_unlock_irq(&conf->resync_lock);
>> }
> I partially agree. The fix that you suggest makes proper accounting of
> whether the array is considered frozen or not.
> But the problem is that even with this fix, both threads will think
> that they "own" the array. And both will do things, like writing data
> or so, which might interfere. The proper fix would ensure that only
> one thread "owns" the array, and the second thread waits until the
> first calls unfreeze_array(), and then the second thread becomes
> "owner" of the array. And of course while there are threads that want
> to "freeze" the array, other threads that want to do raise_barrier
> etc, should wait.
Is that really a problem?
A call to "freeze_array()" doesn't mean "I want to own the array", but
rather "No regular IO should be happening now".
Most callers of freeze_array():
raid1_add_disk(), raid1_remove_disk(), stop(), raid1_reshape()
are called with the "mddev_lock()" mutex held, so they cannot interfere
with each other.
handle_read_error() is called with one pending request, which will block
any call on freeze_array(mddev, 0); - handle_read_error() itself calls
freeze_array(mddev, 1) so it gets access.
So these are already locked against the first four (as lock that
->array_frozen doesn't get corrupted).
That just leaves raid1_quiesce().
That is mostly called under mddev_lock() so it won't interfere with the
others.
The one exception is where md_do_sync() calls
mddev->pers->quiesce(mddev, 1);
mddev->pers->quiesce(mddev, 0);
As this doesn't "claim" the array, but just needs to ensure all pending
IO completes, I don't think there is a problem.
So it seems to me that your concerns are not actually a problem.
Did I miss something?
>
> I am sorry for giving two negative responses in one email:)
Better a negative response than no response :-)
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply related
* Re: Recover filenames from failed RAID0
From: NeilBrown @ 2016-07-15 4:07 UTC (permalink / raw)
To: Michel Dubois, linux-raid
In-Reply-To: <CAAVC+N5m74MyDqZamHgy71vRMhcEY8w_OkiG-GMQKU=JG-QkxA@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 5757 bytes --]
On Mon, Jul 11 2016, Michel Dubois wrote:
> Dear linux-raid mailing list,
>
> I have a RAID0 array of four 3TB disks that failed on the "third" disk.
Posting to the list is sensible. Posting to lots of over vaguely
related people like you did is not appreciated.
>
> I am aware of the non-redundancy of RAID0 but I would like to recover the
> filenames from that RAID0. If I could recover some data it would be a
> bonus.
If you lose one drive from a RAID0, you have effectively lost
everything. This includes file names and data.
An expert in filesystem forensics might be able to recover some bits,
but it would be a lot of work very fairly little expected reward.
>
> Below you'll find the outputs of the following commands
> mdadm --examine /dev/sd[abcd]1
> fdisk -l
>
> where sda1, sdb1, sdc1 and sdd1 should be the 4 RAID devices.
The information you provide describes a 4-drive RAID1, not RAID0.
All the data should equally be on all devices.
Can't you just
mount /dev/md0 /mnt
and access the files?
What does "fsck -n /dev/md0" report?
what sort of filesystem did you have on /dev/md0??
NeilBrown
>
> What could be my next step?
>
> I thank you for your time
>
> Michel Dubois
>
> ======================
> mdadm --examine /dev/sd[abcd]1
> /dev/sda1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 7d247a6e:7b5d46c8:f52d9c89:db304b21
> Creation Time : Mon Apr 23 19:55:36 2012
> Raid Level : raid1
> Used Dev Size : 20980800 (20.01 GiB 21.48 GB)
> Array Size : 20980800 (20.01 GiB 21.48 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Mon Jun 27 21:12:23 2016
> State : clean
> Active Devices : 3
> Working Devices : 3
> Failed Devices : 1
> Spare Devices : 0
> Checksum : 1a57db60 - correct
> Events : 164275
>
>
> Number Major Minor RaidDevice State
> this 0 8 1 0 active sync /dev/sda1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 0 0 2 faulty removed
> 3 3 8 33 3 active sync /dev/sdc1
> /dev/sdb1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 7d247a6e:7b5d46c8:f52d9c89:db304b21
> Creation Time : Mon Apr 23 19:55:36 2012
> Raid Level : raid1
> Used Dev Size : 20980800 (20.01 GiB 21.48 GB)
> Array Size : 20980800 (20.01 GiB 21.48 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Mon Jun 27 21:12:23 2016
> State : clean
> Active Devices : 3
> Working Devices : 3
> Failed Devices : 1
> Spare Devices : 0
> Checksum : 1a57db72 - correct
> Events : 164275
>
>
> Number Major Minor RaidDevice State
> this 1 8 17 1 active sync /dev/sdb1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 0 0 2 faulty removed
> 3 3 8 33 3 active sync /dev/sdc1
> /dev/sdc1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 7d247a6e:7b5d46c8:f52d9c89:db304b21
> Creation Time : Mon Apr 23 19:55:36 2012
> Raid Level : raid1
> Used Dev Size : 20980800 (20.01 GiB 21.48 GB)
> Array Size : 20980800 (20.01 GiB 21.48 GB)
> Raid Devices : 4
> Total Devices : 3
> Preferred Minor : 0
>
> Update Time : Mon Jun 27 21:12:23 2016
> State : clean
> Active Devices : 3
> Working Devices : 3
> Failed Devices : 1
> Spare Devices : 0
> Checksum : 1a57db86 - correct
> Events : 164275
>
>
> Number Major Minor RaidDevice State
> this 3 8 33 3 active sync /dev/sdc1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 0 0 2 faulty removed
> 3 3 8 33 3 active sync /dev/sdc1
>
> ======================
> fdisk -l
>
> WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk
> doesn't support GPT. Use GNU Parted.
>
>
> Disk /dev/sda: 3000.5 GB, 3000592982016 bytes
> 255 heads, 63 sectors/track, 364801 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x03afffbe
>
> Device Boot Start End Blocks Id System
> /dev/sda1 1 267350 2147483647+ ee EFI GPT
>
> WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk
> doesn't support GPT. Use GNU Parted.
>
>
> Disk /dev/sdb: 3000.5 GB, 3000592982016 bytes
> 255 heads, 63 sectors/track, 364801 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x142a889c
>
> Device Boot Start End Blocks Id System
> /dev/sdb1 1 267350 2147483647+ ee EFI GPT
>
> WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk
> doesn't support GPT. Use GNU Parted.
>
>
> Disk /dev/sdc: 3000.5 GB, 3000592982016 bytes
> 255 heads, 63 sectors/track, 364801 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Disk identifier: 0x3daebd50
>
> Device Boot Start End Blocks Id System
> /dev/sdc1 1 267350 2147483647+ ee EFI GPT
>
> Disk /dev/md0: 21.4 GB, 21484339200 bytes
> 2 heads, 4 sectors/track, 5245200 cylinders
> Units = cylinders of 8 * 512 = 4096 bytes
> Disk identifier: 0x00000000
>
> Disk /dev/md0 doesn't contain a valid partition table
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]
^ permalink raw reply
* Re: Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage) with check/repair/sync
From: Matthias Dahl @ 2016-07-15 7:11 UTC (permalink / raw)
To: Michal Hocko
Cc: Tetsuo Handa, linux-raid, linux-mm, dm-devel, linux-kernel,
Mike Snitzer
In-Reply-To: <9074e82f-bf52-011e-8bd7-5731d2b0dcaa@I-love.SAKURA.ne.jp>
Hello...
I am rather persistent (stubborn?) when it comes to tracking down bugs,
if somehow possible... and it seems it paid off... somewhat. ;-)
So I did quite a lot more further tests and came up with something very
interesting: As long as the RAID is in sync (as-in: sync_action=idle),
I can not for the life of me trigger this issue -- the used memory
still explodes to most of the RAM but it oscillates back and forth.
I did very stupid things to stress the machine while dd was running as
usual on the dm-crypt device. I opened a second dd instance with the
same parameters on the dm-crypt device. I wrote a simple program that
allocated random amounts of memory (up to 10 GiB), memset them and after
a random amount of time released it again -- in a continuous loop. I
put heavy network stress on the machine... whatever I could think of.
No matter what, the issue did not trigger. And I repeated said tests
quite a few times over extended time periods (usually an hour or so).
Everything worked beautifully with nice speeds and no noticeable system
slow-downs/lag.
As soon as I issued a "check" to sync_action of the RAID device, it was
just a matter of a second until the OOM killer kicked in and all hell
broke loose again. And basically all of my tests where done while the
RAID device was syncing -- due to a very unfortunate series of events.
I tried to repeat that same test with an external (USB3) connected disk
with a Linux s/w RAID10 over two partitions... but unfortunately that
behaves rather differently. I assume it is because it is connected
through USB and not SATA. While doing those tests on my RAID10 with the
4 internal SATA3 disks, you can see w/ free that the "used memory" does
explode to most of the RAM and then oscillates back and forth. With the
same test on the external disk through, that does not happen at all. The
used memory stays pretty much constant and only the buffers vary... but
most of the memory is still free in that case.
I hope my persistence on the matter is not annoying and finally leads us
somewhere where the real issue hides.
Any suggestions, opinions and ideas are greatly appreciated as I have
pretty much exhausted mine at this time.
Last but not least: I switched my testing to a OpenSuSE Tumbleweed Live
system (x86_64 w/ kernel 4.6.3) as Rawhide w/ 4.7.0rcX behaves rather
strangely and unstable at times.
Thanks,
Matthias
--
Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu
services: custom software [desktop, mobile, web], server administration
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* [PATCH] md: Prevent IO hold during accessing to failed raid5 array
From: Alexey Obitotskiy @ 2016-07-15 13:24 UTC (permalink / raw)
To: shli; +Cc: linux-raid
After array enters in failed state (e.g. number of failed drives
becomes more then accepted for raid5 level) it sets error flags
(one of this flags is MD_CHANGE_PENDING). This flag prevents to
finish all new or non-finished IOs to array and hold them in
pending state. In some cases this can leads to deadlock situation.
For example udev handle array state changes (drives becomes faulty)
and blkid started but unable to finish reads due to IO hold.
At the same time we unable to get exclusive access to array
(to stop array in our case) because another external application
still use this array (blkid in our case).
Fix makes possible to return IO with errors immediately.
So external application can finish working with array and
give exclusive access to other applications.
Signed-off-by: Alexey Obitotskiy <aleksey.obitotskiy@intel.com>
---
drivers/md/raid5.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index 6c1149d..99471b6 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -4692,7 +4692,9 @@ finish:
}
if (!bio_list_empty(&s.return_bi)) {
- if (test_bit(MD_CHANGE_PENDING, &conf->mddev->flags)) {
+ if (test_bit(MD_CHANGE_PENDING, &conf->mddev->flags) &&
+ (s.failed <= conf->max_degraded ||
+ conf->mddev->external == 0)) {
spin_lock_irq(&conf->device_lock);
bio_list_merge(&conf->return_bi, &s.return_bi);
spin_unlock_irq(&conf->device_lock);
--
2.7.4
^ permalink raw reply related
* [PATCH] dm: fix parameter to blk_delay_queue()
From: Tahsin Erdogan @ 2016-07-15 13:27 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, Jens Axboe, dm-devel, Shaohua Li,
linux-raid
Cc: linux-kernel, Tahsin Erdogan
Second parameter to blk_delay_queue() must be in msec units not jiffies.
Signed-off-by: Tahsin Erdogan <tahsin@google.com>
---
drivers/md/dm.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 1b2f96205361..17c63265a205 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -2175,7 +2175,7 @@ static void dm_request_fn(struct request_queue *q)
md_in_flight(md) && rq->bio && rq->bio->bi_vcnt == 1 &&
md->last_rq_pos == pos && md->last_rq_rw == rq_data_dir(rq)) ||
(ti->type->busy && ti->type->busy(ti))) {
- blk_delay_queue(q, HZ / 100);
+ blk_delay_queue(q, 10);
return;
}
--
2.8.0.rc3.226.g39d4020
^ permalink raw reply related
* Re: dm: fix parameter to blk_delay_queue()
From: Mike Snitzer @ 2016-07-15 13:59 UTC (permalink / raw)
To: Tahsin Erdogan
Cc: Alasdair Kergon, Jens Axboe, dm-devel, Shaohua Li, linux-raid,
linux-kernel
In-Reply-To: <1468589228-20722-1-git-send-email-tahsin@google.com>
On Fri, Jul 15 2016 at 9:27am -0400,
Tahsin Erdogan <tahsin@google.com> wrote:
> Second parameter to blk_delay_queue() must be in msec units not jiffies.
>
> Signed-off-by: Tahsin Erdogan <tahsin@google.com>
This needs to be rebased against linux-next (or linux-dm.git's
'for-next') because the code in question has been moved out to dm-rq.c
But I'll gladly take care of it.
Your change seems fine, and in fact my intent was always to have it be
10 msec, see commit: d548b34b062b60
Curious that the initial "HZ / 10" (from Jens' commit 7eaceaccab) wasn't
expressed in terms of msecs either.
Anyway, thanks for the fix!
Mike
> ---
> drivers/md/dm.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/md/dm.c b/drivers/md/dm.c
> index 1b2f96205361..17c63265a205 100644
> --- a/drivers/md/dm.c
> +++ b/drivers/md/dm.c
> @@ -2175,7 +2175,7 @@ static void dm_request_fn(struct request_queue *q)
> md_in_flight(md) && rq->bio && rq->bio->bi_vcnt == 1 &&
> md->last_rq_pos == pos && md->last_rq_rw == rq_data_dir(rq)) ||
> (ti->type->busy && ti->type->busy(ti))) {
> - blk_delay_queue(q, HZ / 100);
> + blk_delay_queue(q, 10);
> return;
> }
>
> --
> 2.8.0.rc3.226.g39d4020
>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox