From: NeilBrown <neilb@suse.de>
To: Milan Broz <mbroz@redhat.com>
Cc: linux-raid@vger.kernel.org
Subject: Re: md raid5 fsync deadlock
Date: Sun, 4 Mar 2012 20:20:13 +1100 [thread overview]
Message-ID: <20120304202013.78a2f65c@notabene.brown> (raw)
In-Reply-To: <4F4F3753.80505@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1366 bytes --]
On Thu, 01 Mar 2012 09:46:11 +0100 Milan Broz <mbroz@redhat.com> wrote:
> On 03/01/2012 02:53 AM, NeilBrown wrote:
> > On Thu, 01 Mar 2012 00:31:08 +0100 Milan Broz<mbroz@redhat.com> wrote:
>
> > Are you certain it is a deadlock? No forward progress at all?
>
> Seems so, it was for several hours in this state without progress.
>
> > What is in md/stripe_cache_size? Does it change?
>
> > What happens if you double the number in stripe_cache_size? What if you
> > double it again?
>
> stripe_cache_size was 256, I doubled it to 512, now
> stripe_cache_active is 390
> stripe_cache size is 512
> and no progress.
>
> With stripe_cache size 1024 it survived few iterations of fio run, now it is
> locked up again:
> stripe_cache_active is 921
> stripe_cache size is 1024
>
That definitely looks like something getting stuck inside RAID5. There are
390 (or 921) stripes that should be being processed but they are blocked
waiting for something.
I would suggest modifying the 'status' function in raid5.c to print out some
details about the stripes in the stripe cache.
You would need to spinlock device_lock, then walk through each chain from
stripe_hashtbl and print out the 'state' and 'count' for each stripe-head and
flags and various bio pointers from each dev.
That might be helpful.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
prev parent reply other threads:[~2012-03-04 9:20 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-02-29 23:31 md raid5 fsync deadlock Milan Broz
2012-03-01 1:53 ` NeilBrown
2012-03-01 8:46 ` Milan Broz
2012-03-04 9:20 ` NeilBrown [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120304202013.78a2f65c@notabene.brown \
--to=neilb@suse.de \
--cc=linux-raid@vger.kernel.org \
--cc=mbroz@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).