From: Dave Chinner <david@fromorbit.com>
To: "Earl, Joshua P" <Joshua.Earl@DrexelMed.edu>
Cc: Brian Foster <bfoster@redhat.com>, "xfs@oss.sgi.com" <xfs@oss.sgi.com>
Subject: Re: xfsxyncd in 'D' state
Date: Fri, 18 Sep 2015 12:03:51 +1000 [thread overview]
Message-ID: <20150918020351.GS3902@dastard> (raw)
In-Reply-To: <2CC86DBF85FEEC41A2DFE1647B40613D5DAF2DFC@NCB-SV-117.DUCOM.edu>
On Thu, Sep 17, 2015 at 09:37:09PM +0000, Earl, Joshua P wrote:
> Hi Brian,
>
> Sorry about the top posting thing... I'm not sure how to control
> that, is my replying somehow messing with that?
when everthing is backwards
to read the thread
it makes it hard
And please wrap your text at 72 columns.
> With good news, I seem to have figured out what was going on. I
> had a cron job which would run every 15 minutes which changed the
> permissions in a directory:
> chmod -R g+rwx /data/shared/homes/bjanto/*
> chmod -R g+rwx /data/shared/homes/lanastor/*
> chgrp -hR ilmn /data/nextseq/*
> chgrp -hR lab /data/shared/homes/*
So you are modifying a large amount of metadata every 15 minutes,
and then you have a problem with your 22-disk wide RAID6 array when
the metadata gets written back. metadata writeback is, by the nature
of metadata in a filesystem, done in small, isolated IOs that cause
large RAID5/6 arrays to do an stripe-wide RMW cycle on every IO.
> > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> > sda 0.29 3.61 5.78 3.58 0.10 0.03 28.27 0.05 5.19 2.39 2.24
> > sdb 1.02 8.66 31.50 3.91 0.33 0.12 26.14 5.94 167.54 27.47 97.25
> >
> > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> > sda 0.00 1.60 0.00 2.00 0.00 0.01 14.40 0.01 4.30 4.30 0.86
> > sdb 0.00 0.00 0.00 0.80 0.00 0.03 64.00 6.46 6332.75 1250.00 100.00
That's pretty clear that your hardware raid array is taking over a
second per IO that requires a RMW cycle. So not a filesystem
problem...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
next prev parent reply other threads:[~2015-09-18 2:08 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-09-16 16:50 xfsxyncd in 'D' state Earl, Joshua P
2015-09-17 16:45 ` Earl, Joshua P
2015-09-17 19:21 ` Brian Foster
2015-09-17 21:37 ` Earl, Joshua P
2015-09-18 2:03 ` Dave Chinner [this message]
2015-09-18 23:51 ` Earl, Joshua P
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150918020351.GS3902@dastard \
--to=david@fromorbit.com \
--cc=Joshua.Earl@DrexelMed.edu \
--cc=bfoster@redhat.com \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox