Re: XFS IO multiplication problem on centos/rhel 6 using hp p420i raid controllers

Dennis Kaarsemaker
2013-03-08T11:48:39Z

On Fri, 2013-03-08 at 10:09 +0100, Dennis Kaarsemaker wrote:
> On Fri, 2013-03-08 at 10:00 +1100, Dave Chinner wrote:
> > On Thu, Mar 07, 2013 at 11:12:08AM +0100, Dennis Kaarsemaker wrote:
> > > On Thu, 2013-03-07 at 14:57 +1100, Dave Chinner wrote:
> > > > On Wed, Mar 06, 2013 at 02:53:12PM +0100, Dennis Kaarsemaker wrote:
> > ....
> > > > > #<----CPU[HYPER]-----><----------Disks-----------><----------Network---------->
> > > > > #cpu sys inter ctxsw KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut
> > > > > 1 0 1636 4219 16 1 2336 313 184 195 12 133
> > > > > 1 0 1654 2804 64 3 2919 432 391 352 20 208
> > > > >
> > > > > [root@bc291bprdb-01 ~]# collectl
> > > > > #<----CPU[HYPER]-----><----------Disks-----------><----------Network---------->
> > > > > #cpu sys inter ctxsw KBRead Reads KBWrit Writes KBIn PktIn KBOut PktOut
> > > > > 1 0 2220 3691 332 13 39992 331 112 122 6 92
> > > > > 0 0 1354 2708 0 0 39836 335 103 125 9 99
> > > > > 0 0 1563 3023 120 6 44036 369 399 317 13 188
> > > > >
> > > > > Notice the KBWrit difference. These are two identical hp gen 8 machines,
> > > > > doing the same thing (replicating the same mysql schema). The one
> > > > > writing ten times as many bytes in the same amount of transactions is
> > > > > running centos 6 (and was running rhel 6).
> > > >
> > > > So what is the problem? it is writing too much on the on the centos
> > > > 6 machine? Either way, this doesn't sound like a filesystem problem
> > > > - the size and amount of data writes is entirely determined by the
> > > > application.
> > >
> > > For performing the same amount of work (processing the same mysql
> > > transactions, the same amount of IO transactions resulting from them),
> > > the 'broken' case writes ten-ish times as many bytes.
> >
> > Thanks for clarifying.
> >
> > > > > /dev/mapper/sysvm-mysqlVol /mysql/bp xfs rw,relatime,attr2,delaylog,allocsize=1024k,logbsize=256k,sunit=512,swidth=1536,noquota 0 0
> > > >
> > > > What is the reason for using allocsize, sunit/swidth? Are you using
> > > > them on other machines?
> > >
> > > xfs autodetects them from the hpsa driver. They seem to be correct for
> > > the raid layout (256 strips, 3 drives per mirror pool) and I don't seem
> > > to be able to override them.
> >
> > That's fine, they're set correctly. I'd forgotten that the number
> > are emitted in /proc/mounts even when they are not specified as
> > mount options.
> >
> > > > And if you remove the allocsize mount option, does the behaviour on
> > > > centos6.3 change? What happens if you set allocsize=4k?
> > >
> > > The allocsize parameter has no effect. It was put in place to correct a
> > > monitoring issue: due to mysql's access patterns, using the default
> > > large allocsize on rhel 6 makes our monitoring report the filesystem as
> > > much fuller than it actually is.
> >
> > Which is due to speculative EOF preallocation, and so it is only set
> > on the CentOS box that is showing the larger write behaviour? Have
> > you tried setting it to 4k? If not, please do - EOF preallocation for
> > sparse extending writes can result in extra zeroing occurring, and
> > so if it is anything related to the filesystem, this is the likely
> > culprit. Setting it to 4k sets it back to the default value used
> > on older versions of Linux....
>
> I've set it to 4k, but no change, though I haven't rebuilt the files yet
> with this setting (doing that as we speak, takes 90 minutes). I'm also
> wondering how this could cause the increasing bytes out as reported by
> vmstat, should zeroing do that?

Unfortunately, even on a rebuilt filesystem, the symptoms did not
change.