Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Josef Bacik <josef@toxicpanda.com>]

On Wed, Mar 09, 2022 at 10:41:53AM -0800, Luis Chamberlain wrote:
> On Tue, Mar 08, 2022 at 11:40:18AM -0500, Theodore Ts'o wrote:
> > One of my team members has been working with Darrick to set up a set
> > of xfs configs[1] recommended by Darrick, and she's stood up an
> > automated test spinner using gce-xfstests which can watch a git branch
> > and automatically kick off a set of tests whenever it is updated.
> 
> I think its important to note, as we would all know, that contrary to
> most other subsystems, in so far as blktests and fstests is concerned,
> simply passing a test once does not mean there is no issue given that
> some test can fail with a failure rate of 1/1,000 for instance.
> 

FWIW we (the btrfs team) have been running nightly runs of fstests against our
devel branch for over a year and tracking the results.  This allowed us to get
down to 0 failures because we could identify flakey tests and fix them or simply
disable them.  Then this means when we do have one of those 1/1,000 failures in
one of our configs (I copied Ted's approach and test all the various feature
combos) we know what set of parameters the failure was on and can go run that
test in a loop to reproduce the problem.

We like this approach because it's not a "wait a week to see if something
failed", we know the day after some new thing was merged if it caused a problem.
If it's more subtle then we still find it because a test will start failing at
some point.  It's a nice balance in how long we have to wait for results and
allows us to be a lot more sure in merging new code without hemming and hawing
for months.  Thanks,

Josef