Re: Regression ~7% write, 3.17.1 vs 3.18.0rc5

[Dave Chinner <david@fromorbit.com>]

On Fri, Nov 28, 2014 at 06:45:37PM +0100, Matus Kocka wrote:
> Hello,
> 
> we see the ~7% regression in the initial WRITE on xfs in in-cache
> mode by IOZONE test
> between 3.17.1-306.el7 and  3.18.0-0.rc5.git0.1.el7 (respectively
> 3.18.0-0.rc3.git0.2.el7)
> 
> By in-cache mode we mean that file fits into the OS cache.

As we all know by now, this iozone test is extremely sensitive to
changes in CPU cache footprint rather than anything to do with the
IO subsystem. Is this "regression" reproducable on more than one
machine?  Chasing IOZone "regressions" measured on a single machine
is a wild goose chase....

> For big file-sizes >4GiB (out of cache mode) there is no performance drop.

So it's a CPU usage issue, then?

> [Files: https://www.dropbox.com/sh/2sx0ejt73n4idcp/AAA0nYhZTDAMGigJ-B5yqTwga?dl=0]
> 
> 1, Firstly, (xfs_SATA.pdf) we see the ~7% regression in the in-cache
> IOZONE test on 1 TB SATA disk with cfq io-elevator,
> 
> 2, We also see it (xfs_write.pdf) using same in-cache test on the
> fast pci SSD disk (RealSSD P320h 350GB)
> 
> 3, We have tried direct-io tests on the same pci SSD with no
> regression (directio.pdf)
> 
> It seems that this regression is related to cache since DIRECT IO
> and out-of-cache tests
>   do not have the regression. Also, rewrite test does not show any
> regression.
> 
> Would somebody from the developers be interested to look into it and
> try to hunt down this bug?

Can you please bisect so we can identify the problematic commit?

> We can provide more details regarding our testing methodology but we
> have neither capacity nor
> expertise to debug it ourselves.

You don't need to debug the problem, just run the test repeatedly on
the kernels the bisect builds until it identifies the commit that
introduced the regression.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com