Re: [LSF/MM TOPIC] Filesystem performance regression tests

[Dave Chinner <david@fromorbit.com>]

On Thu, Feb 01, 2018 at 09:21:40AM -0500, Josef Bacik wrote:
> On Thu, Feb 01, 2018 at 08:50:21AM +0200, Amir Goldstein wrote:
> > Hi Josef and all,
> > 
> > I would like to be rude and solicit a talk from Josef on automated
> > performance regression testing (if he is planning to attend).
> > We know the guys at facebook are running some performance
> > regression tests for a while and Josef has just upstreamed
> > a nice taste on this infrastructure to xfstests:
> > https://marc.info/?l=fstests&m=150765617921864&w=2
> > 
> > But this is only the beginning... for community performance
> > regression tests to be useful there need to be not only people
> > running the tests, but also people running the tests on well known
> > machines and/or well known hardware configurations and maintain
> > long lived performance results db for those machines.
> > 
> > How can we utilize community resources to achieve that?
> > Can running performance regressions of gce-xfstests provide
> > anything close to stable results?
> > 
> > If performance regressions are integrated into 0-day kernel test
> > robot, that could be extremely beneficial to the community, but can
> > the robot guaranty to run the tests on dedicated machines or
> > VMs with dedicated resources?
> > 
> > Do we know of good examples to follow from automated regression
> > tests done for specific filesystem (Dave Chinner has referred to his
> > regression tests in one or two occasions)? for other kernel subsystems?
> > 
> > Putting up a regression test server for overlayfs is on my TODO list.
> > In the mean while, I have little to contribute from my experience, but
> > would love to sit in that talk.
> > 
> 
> I'm happy to talk about this stuff and how it could be improved.  I think
> integrating it into continuous testing is tricky.  With all of our performance
> stuff it's always A/B testing, because shit changes constantly.  I hate doing
> perf stuff in VM's because it's captive to whatever else the host is doing.
> xfstests is a good place for this stuff since we all have our own personal rigs
> that we control.  Could we extend xfstests to log our results publicly?  That
> would be cool, I would be fine with that.

I recently wrote a script that produces html comparison tables from
multiple test result runs for easy viewing of long term failure
trends.  I could probably adapt it to whatever the perf test results
output, too. The main problem is where to put them online - perhaps
a git repo somewhere we can all commit to that auto updates to a web
server?

> My test box rarely changes, so if its
> just a matter of uploading some magic ID associated with my box, and then
> uploading results paired with that ID to be world viewed then that would be
> cool.  I'm sure we could find somebody willing to host such a thing for us.

Write the script and they will come? :P

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com