Re: [LSF/MM TOPIC] Filesystem performance regression tests

[Josef Bacik <josef@toxicpanda.com>]

On Thu, Feb 01, 2018 at 08:50:21AM +0200, Amir Goldstein wrote:
> Hi Josef and all,
> 
> I would like to be rude and solicit a talk from Josef on automated
> performance regression testing (if he is planning to attend).
> We know the guys at facebook are running some performance
> regression tests for a while and Josef has just upstreamed
> a nice taste on this infrastructure to xfstests:
> https://marc.info/?l=fstests&m=150765617921864&w=2
> 
> But this is only the beginning... for community performance
> regression tests to be useful there need to be not only people
> running the tests, but also people running the tests on well known
> machines and/or well known hardware configurations and maintain
> long lived performance results db for those machines.
> 
> How can we utilize community resources to achieve that?
> Can running performance regressions of gce-xfstests provide
> anything close to stable results?
> 
> If performance regressions are integrated into 0-day kernel test
> robot, that could be extremely beneficial to the community, but can
> the robot guaranty to run the tests on dedicated machines or
> VMs with dedicated resources?
> 
> Do we know of good examples to follow from automated regression
> tests done for specific filesystem (Dave Chinner has referred to his
> regression tests in one or two occasions)? for other kernel subsystems?
> 
> Putting up a regression test server for overlayfs is on my TODO list.
> In the mean while, I have little to contribute from my experience, but
> would love to sit in that talk.
> 

I'm happy to talk about this stuff and how it could be improved.  I think
integrating it into continuous testing is tricky.  With all of our performance
stuff it's always A/B testing, because shit changes constantly.  I hate doing
perf stuff in VM's because it's captive to whatever else the host is doing.
xfstests is a good place for this stuff since we all have our own personal rigs
that we control.  Could we extend xfstests to log our results publicly?  That
would be cool, I would be fine with that.  My test box rarely changes, so if its
just a matter of uploading some magic ID associated with my box, and then
uploading results paired with that ID to be world viewed then that would be
cool.  I'm sure we could find somebody willing to host such a thing for us.
Thanks,

Josef