From: NeilBrown <neilb@suse.de>
To: Jakob Unterwurzacher <jakobunt@gmail.com>
Cc: linux-raid@vger.kernel.org
Subject: Re: md raid6: slow resync if fs is mounted rw
Date: Wed, 26 Feb 2014 10:38:01 +1100 [thread overview]
Message-ID: <20140226103801.72e24304@notabene.brown> (raw)
In-Reply-To: <530D1F53.5070703@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2047 bytes --]
On Tue, 25 Feb 2014 23:55:15 +0100 Jakob Unterwurzacher <jakobunt@gmail.com>
wrote:
> Dear list,
>
> I am seeing a strange performance issue on a 4 x 4TB md raid6 on kernel
> 3.12. Initial resync goes nice and fast at 130MB/s:
>
> > # cat /proc/mdstat
> > Personalities : [raid1] [raid6] [raid5] [raid4]
> > md1 : active raid6 sda3[0] sdd3[3] sdc3[2] sdb3[1]
> > 7751269376 blocks super 1.2 level 6, 512k chunk, algorithm 2 [4/4] [UUUU]
> > [=>...................] resync = 7.5% (293925504/3875634688) finish=458.4min speed=130210K/sec
> >
> > md0 : active raid1 sda2[0] sdd2[3] sdc2[2] sdb2[1]
> > 29280128 blocks super 1.2 [4/4] [UUUU]
> >
> > unused devices: <none>
>
> ...unless the ext4 filesystem on the array is mounted read-write!
> Then, speed plummets to 1 - 7MB/s. The filesystem is empty and totally
> idle; a simple "mount -o remount,ro" restores the speed to 130MB/s.
>
> I dumped the /sys/block/md1/md tree with the filesystem mounted ro and
> rw, the diff looks like this:
>
> > # diff -u md.rw md.ro
> > [...]
> > --- md.rw/stripe_cache_active 2014-02-25 17:33:31.250019324 +0100
> > +++ md.ro/stripe_cache_active 2014-02-25 17:33:31.250019324 +0100
> > @@ -1 +1 @@
> > -0
> > +256
> > diff -u md.rw/sync_speed md.ro/sync_speed
> > --- md.rw/sync_speed 2014-02-25 17:33:31.250019324 +0100
> > +++ md.ro/sync_speed 2014-02-25 17:33:31.250019324 +0100
> > @@ -1 +1 @@
> > -1052
> > +126244
>
> So, stripe_cache_active is zero if the fs is mounted rw. Hmm.
>
> Interestingly, if I increase sync_speed_min to 100MB/s, I do get 100MB/s.
>
> Any idea what is going on?
>
Simplest explanation would be that ext4 is writing to the array
occasionally. Maybe 1 block per second. That would be enough for md to
decide that the array is busy and to back-off the resync.
Try
blktrace -w 20 /dev/md1
while the filesystem is mounted read-write, then
blkparse md1.blktrace*
do we if anything writes are going to the array.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
next prev parent reply other threads:[~2014-02-25 23:38 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-02-25 22:55 md raid6: slow resync if fs is mounted rw Jakob Unterwurzacher
2014-02-25 23:38 ` NeilBrown [this message]
2014-02-26 2:31 ` Brad Campbell
2014-02-26 9:40 ` Jakob Unterwurzacher
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140226103801.72e24304@notabene.brown \
--to=neilb@suse.de \
--cc=jakobunt@gmail.com \
--cc=linux-raid@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).