From: David Dillow <dave@thedillows.org>
To: Kevin Ross <kevin@familyross.net>
Cc: Phil Turmel <philip@turmel.org>,
linux-kernel@vger.kernel.org,
linux-raid <linux-raid@vger.kernel.org>
Subject: Re: RAID extremely slow
Date: Thu, 26 Jul 2012 22:15:03 -0400 [thread overview]
Message-ID: <1343355303.29938.5.camel@obelisk.thedillows.org> (raw)
In-Reply-To: <5010A386.4080209@familyross.net>
On Wed, 2012-07-25 at 18:55 -0700, Kevin Ross wrote:
> On 07/25/2012 06:00 PM, Phil Turmel wrote:
> > Piles of small reads scattered across multiple drives, and a
> > concentration of queued writes to /dev/sda. What's on /dev/sda?
> > It's not a member of the raid, so it must be some other system task
> > involved.
> After rebooting, MythTV is currently recording two shows, and the resync
> is running at full speed.
>
> # cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4]
> md0 : active raid6 sdh1[0] sdd1[9] sde1[10] sdb1[6] sdi1[7] sdc1[4]
> sdf1[3] sdg1[8] sdj1[1]
> 6837311488 blocks super 1.2 level 6, 512k chunk, algorithm 2
> [9/9] [UUUUUUUUU]
> [=>...................] resync = 9.3% (91363840/976758784)
> finish=1434.3min speed=10287K/sec
>
> unused devices: <none>
>
> atop shows the avio of all the drives to be less than 1ms, where before
> they were much higher. It will run for a couple days under load just
> fine, and then it will come to a halt.
>
> It's a 3.2.21 kernel. I'm running Debian Testing, and the exact Debian
> package version is:
I suspect you are being hit by same bug I was -- delayed stripes never
got processed. If you get into the state where the rebuild isn't
progressing, and you find that increasing the size of the stripe cache
allows the rebuild to proceed (but the filesystem stays wedged), then
that cinches it.
If you can, upgrade to the latest 3.4 stable kernel (3.4.6 right now).
As far as I can see, the latest 3.2 stable does not contain the delayed
stripe fix.
After applying those fixes to my kernel, my MythTV setup over a 5 disk
RAID5 has been pretty solid, where before I was getting lockups every
few days. It still seems to be getting slower over time, but I've not
looked into it yet as it is not as catastrophic as the wedging.
HTH,
Dave
next prev parent reply other threads:[~2012-07-27 2:15 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <501078B2.8070707@familyross.net>
2012-07-26 1:00 ` RAID extremely slow Phil Turmel
2012-07-26 1:55 ` Kevin Ross
2012-07-26 2:09 ` CoolCold
2012-07-26 2:18 ` Kevin Ross
2012-07-26 5:00 ` Kevin Ross
2012-07-26 22:36 ` Kevin Ross
2012-07-27 19:08 ` Bill Davidsen
2012-07-27 21:45 ` Kevin Ross
2012-07-28 4:45 ` Grant Coady
2012-07-28 8:34 ` Kevin Ross
2012-08-01 3:16 ` Bill Davidsen
2012-07-27 2:15 ` David Dillow [this message]
2012-07-27 2:17 ` David Dillow
2012-07-27 2:17 ` Kevin Ross
2012-07-27 2:27 ` David Dillow
2012-07-27 2:53 ` Kevin Ross
2012-07-27 3:17 ` Kevin Ross
2012-08-17 21:55 ` Jan Engelhardt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1343355303.29938.5.camel@obelisk.thedillows.org \
--to=dave@thedillows.org \
--cc=kevin@familyross.net \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=philip@turmel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox