Regression ~7% write, 3.17.1 vs 3.18.0rc5

Regression ~7% write, 3.17.1 vs 3.18.0rc5

[Matus Kocka]

Hello,

we see the ~7% regression in the initial WRITE on xfs in in-cache mode 
by IOZONE test
between 3.17.1-306.el7 and  3.18.0-0.rc5.git0.1.el7 (respectively 
3.18.0-0.rc3.git0.2.el7)

By in-cache mode we mean that file fits into the OS cache.
For big file-sizes >4GiB (out of cache mode) there is no performance drop.

[Files: 
https://www.dropbox.com/sh/2sx0ejt73n4idcp/AAA0nYhZTDAMGigJ-B5yqTwga?dl=0]

1, Firstly, (xfs_SATA.pdf) we see the ~7% regression in the in-cache
IOZONE test on 1 TB SATA disk with cfq io-elevator,

2, We also see it (xfs_write.pdf) using same in-cache test on the
fast pci SSD disk (RealSSD P320h 350GB)

3, We have tried direct-io tests on the same pci SSD with no regression 
(directio.pdf)

It seems that this regression is related to cache since DIRECT IO and 
out-of-cache tests
   do not have the regression. Also, rewrite test does not show any 
regression.

Would somebody from the developers be interested to look into it and try 
to hunt down this bug?
We can provide more details regarding our testing methodology but we 
have neither capacity nor
expertise to debug it ourselves.

-- 
Matus Kocka

Quality Assurance Associate
Red Hat Czech s.r.o.
Purkyňova 99/71, 612 45, Brno, Czech Republic

    tel.: +420 532 294 793
direct: 8262793
   email: mkocka [at] redhat.com


--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Regression ~7% write, 3.17.1 vs 3.18.0rc5

[Dave Chinner]

On Fri, Nov 28, 2014 at 06:45:37PM +0100, Matus Kocka wrote:
> Hello,
> 
> we see the ~7% regression in the initial WRITE on xfs in in-cache
> mode by IOZONE test
> between 3.17.1-306.el7 and  3.18.0-0.rc5.git0.1.el7 (respectively
> 3.18.0-0.rc3.git0.2.el7)
> 
> By in-cache mode we mean that file fits into the OS cache.

As we all know by now, this iozone test is extremely sensitive to
changes in CPU cache footprint rather than anything to do with the
IO subsystem. Is this "regression" reproducable on more than one
machine?  Chasing IOZone "regressions" measured on a single machine
is a wild goose chase....

> For big file-sizes >4GiB (out of cache mode) there is no performance drop.

So it's a CPU usage issue, then?

> [Files: https://www.dropbox.com/sh/2sx0ejt73n4idcp/AAA0nYhZTDAMGigJ-B5yqTwga?dl=0]
> 
> 1, Firstly, (xfs_SATA.pdf) we see the ~7% regression in the in-cache
> IOZONE test on 1 TB SATA disk with cfq io-elevator,
> 
> 2, We also see it (xfs_write.pdf) using same in-cache test on the
> fast pci SSD disk (RealSSD P320h 350GB)
> 
> 3, We have tried direct-io tests on the same pci SSD with no
> regression (directio.pdf)
> 
> It seems that this regression is related to cache since DIRECT IO
> and out-of-cache tests
>   do not have the regression. Also, rewrite test does not show any
> regression.
> 
> Would somebody from the developers be interested to look into it and
> try to hunt down this bug?

Can you please bisect so we can identify the problematic commit?

> We can provide more details regarding our testing methodology but we
> have neither capacity nor
> expertise to debug it ourselves.

You don't need to debug the problem, just run the test repeatedly on
the kernels the bisect builds until it identifies the commit that
introduced the regression.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com