[ANNOUNCE] New GitHub CI workflow

[Josef Bacik <josef@toxicpanda.com>]

Hello,

For the last few years we've been relying on my nightly tests to catch any
regressions we have in our development tree.  The results have been crudely
dumped to http://toxicpanda.com and http://toxicpanda.com/performance/.  This is
a bunch of VM's on a Intel NUC with cronjobs and chewing gum holding it all
together.  It has worked pretty well, but it is beginning to show it's age, as
the overnight runs literally take all night, starting around 8PM my time and
finishing 1:30PM my time.

I've upgraded my hardware and re-built the VM setup.  The old setup will
continue so we have a nice baseline of what's going on, but the new setup will
be triggered with GitHub Actions.

# tl;dr

If you are a btrfs developer and are in the https://github.com/btrfs
organization, you can now submit PR's against the "ci" branch in the "linux"
repo.  This will trigger a full fstests run using my VM's, which currently test
7 different configurations, 5 on x86 and 2 on ARM64.  Zoned will be added
shortly, I've just been fighting with the thing to get this far so that part is
still in the pipeline.

It is highly encouraged that if you've got a decently involved patchset that you
take advantage of this system to pre-check your code, either before submitting
it to the list for review or in parallel.

# Details

On every run we pull from our "staging" branch in our copy of the "fstests" repo
in our https://github.com/btrfs repo.  Everybody in the org has commit access to
this, if you want new tests run through the CI please do feel free to commit
them to this branch and then trigger a CI run.

We also pull down the latest "devel" branch from Sterba's btrfs-progs tree.
This will help us catch any output changes or related user space bugs with our
fstests runs, as well as allow us to easily test new fs features.

These runs take around 4.5 hours currently.  There's a 6 hour timeout, so if
there's a deadlock or a panic it'll simply timeout.  I haven't figured out a way
to capture this information for the results so that's a bummer of a limitation
currently.

# Clean runs

In order to make this as easy as possible for us to catch regressions in our own
code, my goal is to make sure the baseline runs of fstests run with 0 failures.
This means we're utilizing exclude files in this GitHub CI setup.  The old setup
will continue to run the full suite so we can make sure we're tracking what
tests are flakey.

Currently x86 is clean, however ARM still has a few iterations to go before I'll
have nailed down all the flakey tests.  In the first few weeks please
investigate failures to make sure they're not real regressions, or check the
existing http://toxicpanda.com list of runs to see if the test already has found
to be flakey.  In our "staging" branch in our "fstests" repo there is EXCLUDE
and EXCLUDE.arm, simply add tests to the relevant file to expunge them from the
CI runs.

# We're just ignoring flakey tests?

For now, yes.  For things that are easier to reason with, for example the
block-tree root feature is incompatible with ~NO_HOLES, patches are preferred to
detect and skip the test for specific incompatibilities.  However for harder to
debug things feel free to simply exclude them.  With this list of excludes we
can then go back when we have time and nail down each of them and get them fixed
up so they're no longer flakey.

# OMFG GITHUB!?!?

Yes, I'm nothing if not lazy, and honestly it took me about a week to work out
all the kinks in this particular system.  We're already on GH, we already have a
lot of processes around using our existing GH org, so I did this with GitHub
actions.  I have 0 interest in debating this particular choice.  If you aren't a
core developer you aren't my target audience.  If you don't want to use the CI
you don't have to, this is simply a convenience offered to those of us who do
this for our job.

# I'm not in the GH Btrfs org/don't have a GH account and want to use the CI?

If you have a sufficiently large patchset you would like run against the CI for
you all you have to do is ask one of the Btrfs developers and we'd be happy to
push a PR for you so you can get your patches tested.

If you would like to be added to the GH Btrfs Org and aren't a current member
please email myself or Dave Sterba and we can decide if you can be added as a
member, but this will be limited to consistent contributors.  We are always
happy to submit PR's on your behalf if you're just looking for some extra
validation of a large contribution.

Thanks,

Josef