From: NeilBrown <neilb@suse.de>
To: Milan Broz <mbroz@redhat.com>
Cc: linux-raid@vger.kernel.org
Subject: Re: md raid5 fsync deadlock
Date: Sun, 4 Mar 2012 20:20:13 +1100 [thread overview]
Message-ID: <20120304202013.78a2f65c@notabene.brown> (raw)
In-Reply-To: <4F4F3753.80505@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 1366 bytes --]
On Thu, 01 Mar 2012 09:46:11 +0100 Milan Broz <mbroz@redhat.com> wrote:
> On 03/01/2012 02:53 AM, NeilBrown wrote:
> > On Thu, 01 Mar 2012 00:31:08 +0100 Milan Broz<mbroz@redhat.com> wrote:
>
> > Are you certain it is a deadlock? No forward progress at all?
>
> Seems so, it was for several hours in this state without progress.
>
> > What is in md/stripe_cache_size? Does it change?
>
> > What happens if you double the number in stripe_cache_size? What if you
> > double it again?
>
> stripe_cache_size was 256, I doubled it to 512, now
> stripe_cache_active is 390
> stripe_cache size is 512
> and no progress.
>
> With stripe_cache size 1024 it survived few iterations of fio run, now it is
> locked up again:
> stripe_cache_active is 921
> stripe_cache size is 1024
>
That definitely looks like something getting stuck inside RAID5. There are
390 (or 921) stripes that should be being processed but they are blocked
waiting for something.
I would suggest modifying the 'status' function in raid5.c to print out some
details about the stripes in the stripe cache.
You would need to spinlock device_lock, then walk through each chain from
stripe_hashtbl and print out the 'state' and 'count' for each stripe-head and
flags and various bio pointers from each dev.
That might be helpful.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
prev parent reply other threads:[~2012-03-04 9:20 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-02-29 23:31 md raid5 fsync deadlock Milan Broz
2012-03-01 1:53 ` NeilBrown
2012-03-01 8:46 ` Milan Broz
2012-03-04 9:20 ` NeilBrown [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120304202013.78a2f65c@notabene.brown \
--to=neilb@suse.de \
--cc=linux-raid@vger.kernel.org \
--cc=mbroz@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.