[ANNOUNCE] New GitHub CI workflow

[ANNOUNCE] New GitHub CI workflow

[Josef Bacik]

Hello,

For the last few years we've been relying on my nightly tests to catch any
regressions we have in our development tree.  The results have been crudely
dumped to http://toxicpanda.com and http://toxicpanda.com/performance/.  This is
a bunch of VM's on a Intel NUC with cronjobs and chewing gum holding it all
together.  It has worked pretty well, but it is beginning to show it's age, as
the overnight runs literally take all night, starting around 8PM my time and
finishing 1:30PM my time.

I've upgraded my hardware and re-built the VM setup.  The old setup will
continue so we have a nice baseline of what's going on, but the new setup will
be triggered with GitHub Actions.

# tl;dr

If you are a btrfs developer and are in the https://github.com/btrfs
organization, you can now submit PR's against the "ci" branch in the "linux"
repo.  This will trigger a full fstests run using my VM's, which currently test
7 different configurations, 5 on x86 and 2 on ARM64.  Zoned will be added
shortly, I've just been fighting with the thing to get this far so that part is
still in the pipeline.

It is highly encouraged that if you've got a decently involved patchset that you
take advantage of this system to pre-check your code, either before submitting
it to the list for review or in parallel.

# Details

On every run we pull from our "staging" branch in our copy of the "fstests" repo
in our https://github.com/btrfs repo.  Everybody in the org has commit access to
this, if you want new tests run through the CI please do feel free to commit
them to this branch and then trigger a CI run.

We also pull down the latest "devel" branch from Sterba's btrfs-progs tree.
This will help us catch any output changes or related user space bugs with our
fstests runs, as well as allow us to easily test new fs features.

These runs take around 4.5 hours currently.  There's a 6 hour timeout, so if
there's a deadlock or a panic it'll simply timeout.  I haven't figured out a way
to capture this information for the results so that's a bummer of a limitation
currently.

# Clean runs

In order to make this as easy as possible for us to catch regressions in our own
code, my goal is to make sure the baseline runs of fstests run with 0 failures.
This means we're utilizing exclude files in this GitHub CI setup.  The old setup
will continue to run the full suite so we can make sure we're tracking what
tests are flakey.

Currently x86 is clean, however ARM still has a few iterations to go before I'll
have nailed down all the flakey tests.  In the first few weeks please
investigate failures to make sure they're not real regressions, or check the
existing http://toxicpanda.com list of runs to see if the test already has found
to be flakey.  In our "staging" branch in our "fstests" repo there is EXCLUDE
and EXCLUDE.arm, simply add tests to the relevant file to expunge them from the
CI runs.

# We're just ignoring flakey tests?

For now, yes.  For things that are easier to reason with, for example the
block-tree root feature is incompatible with ~NO_HOLES, patches are preferred to
detect and skip the test for specific incompatibilities.  However for harder to
debug things feel free to simply exclude them.  With this list of excludes we
can then go back when we have time and nail down each of them and get them fixed
up so they're no longer flakey.

# OMFG GITHUB!?!?

Yes, I'm nothing if not lazy, and honestly it took me about a week to work out
all the kinks in this particular system.  We're already on GH, we already have a
lot of processes around using our existing GH org, so I did this with GitHub
actions.  I have 0 interest in debating this particular choice.  If you aren't a
core developer you aren't my target audience.  If you don't want to use the CI
you don't have to, this is simply a convenience offered to those of us who do
this for our job.

# I'm not in the GH Btrfs org/don't have a GH account and want to use the CI?

If you have a sufficiently large patchset you would like run against the CI for
you all you have to do is ask one of the Btrfs developers and we'd be happy to
push a PR for you so you can get your patches tested.

If you would like to be added to the GH Btrfs Org and aren't a current member
please email myself or Dave Sterba and we can decide if you can be added as a
member, but this will be limited to consistent contributors.  We are always
happy to submit PR's on your behalf if you're just looking for some extra
validation of a large contribution.

Thanks,

Josef

Re: [ANNOUNCE] New GitHub CI workflow

[Neal Gompa]

On Mon, Jul 17, 2023 at 3:17 PM Josef Bacik <josef@toxicpanda.com> wrote:
>
> Hello,
>
> For the last few years we've been relying on my nightly tests to catch any
> regressions we have in our development tree.  The results have been crudely
> dumped to http://toxicpanda.com and http://toxicpanda.com/performance/.  This is
> a bunch of VM's on a Intel NUC with cronjobs and chewing gum holding it all
> together.  It has worked pretty well, but it is beginning to show it's age, as
> the overnight runs literally take all night, starting around 8PM my time and
> finishing 1:30PM my time.
>
> I've upgraded my hardware and re-built the VM setup.  The old setup will
> continue so we have a nice baseline of what's going on, but the new setup will
> be triggered with GitHub Actions.
>
> # tl;dr
>
> If you are a btrfs developer and are in the https://github.com/btrfs
> organization, you can now submit PR's against the "ci" branch in the "linux"
> repo.  This will trigger a full fstests run using my VM's, which currently test
> 7 different configurations, 5 on x86 and 2 on ARM64.  Zoned will be added
> shortly, I've just been fighting with the thing to get this far so that part is
> still in the pipeline.
>
> It is highly encouraged that if you've got a decently involved patchset that you
> take advantage of this system to pre-check your code, either before submitting
> it to the list for review or in parallel.
>
> # Details
>
> On every run we pull from our "staging" branch in our copy of the "fstests" repo
> in our https://github.com/btrfs repo.  Everybody in the org has commit access to
> this, if you want new tests run through the CI please do feel free to commit
> them to this branch and then trigger a CI run.
>
> We also pull down the latest "devel" branch from Sterba's btrfs-progs tree.
> This will help us catch any output changes or related user space bugs with our
> fstests runs, as well as allow us to easily test new fs features.
>
> These runs take around 4.5 hours currently.  There's a 6 hour timeout, so if
> there's a deadlock or a panic it'll simply timeout.  I haven't figured out a way
> to capture this information for the results so that's a bummer of a limitation
> currently.
>
> # Clean runs
>
> In order to make this as easy as possible for us to catch regressions in our own
> code, my goal is to make sure the baseline runs of fstests run with 0 failures.
> This means we're utilizing exclude files in this GitHub CI setup.  The old setup
> will continue to run the full suite so we can make sure we're tracking what
> tests are flakey.
>
> Currently x86 is clean, however ARM still has a few iterations to go before I'll
> have nailed down all the flakey tests.  In the first few weeks please
> investigate failures to make sure they're not real regressions, or check the
> existing http://toxicpanda.com list of runs to see if the test already has found
> to be flakey.  In our "staging" branch in our "fstests" repo there is EXCLUDE
> and EXCLUDE.arm, simply add tests to the relevant file to expunge them from the
> CI runs.
>
> # We're just ignoring flakey tests?
>
> For now, yes.  For things that are easier to reason with, for example the
> block-tree root feature is incompatible with ~NO_HOLES, patches are preferred to
> detect and skip the test for specific incompatibilities.  However for harder to
> debug things feel free to simply exclude them.  With this list of excludes we
> can then go back when we have time and nail down each of them and get them fixed
> up so they're no longer flakey.
>
> # OMFG GITHUB!?!?
>
> Yes, I'm nothing if not lazy, and honestly it took me about a week to work out
> all the kinks in this particular system.  We're already on GH, we already have a
> lot of processes around using our existing GH org, so I did this with GitHub
> actions.  I have 0 interest in debating this particular choice.  If you aren't a
> core developer you aren't my target audience.  If you don't want to use the CI
> you don't have to, this is simply a convenience offered to those of us who do
> this for our job.
>
> # I'm not in the GH Btrfs org/don't have a GH account and want to use the CI?
>
> If you have a sufficiently large patchset you would like run against the CI for
> you all you have to do is ask one of the Btrfs developers and we'd be happy to
> push a PR for you so you can get your patches tested.
>
> If you would like to be added to the GH Btrfs Org and aren't a current member
> please email myself or Dave Sterba and we can decide if you can be added as a
> member, but this will be limited to consistent contributors.  We are always
> happy to submit PR's on your behalf if you're just looking for some extra
> validation of a large contribution.
>

Will we move btrfs-progs to this GitHub organization?


-- 
真実はいつも一つ！/ Always, there's only one truth!

Re: [ANNOUNCE] New GitHub CI workflow

[David Sterba]

On Mon, Jul 17, 2023 at 05:14:36PM -0400, Neal Gompa wrote:
> On Mon, Jul 17, 2023 at 3:17 PM Josef Bacik <josef@toxicpanda.com> wrote:
> > you don't have to, this is simply a convenience offered to those of us who do
> > this for our job.
> >
> > # I'm not in the GH Btrfs org/don't have a GH account and want to use the CI?
> >
> > If you have a sufficiently large patchset you would like run against the CI for
> > you all you have to do is ask one of the Btrfs developers and we'd be happy to
> > push a PR for you so you can get your patches tested.
> >
> > If you would like to be added to the GH Btrfs Org and aren't a current member
> > please email myself or Dave Sterba and we can decide if you can be added as a
> > member, but this will be limited to consistent contributors.  We are always
> > happy to submit PR's on your behalf if you're just looking for some extra
> > validation of a large contribution.
> >
> 
> Will we move btrfs-progs to this GitHub organization?

Why?  I can create a repository and push the releases there, but for
development a lot of things would have to be changed to make everything
work, scripts, integration, permissions, etc.

Re: [ANNOUNCE] New GitHub CI workflow

[David Sterba]

On Mon, Jul 17, 2023 at 02:57:15PM -0400, Josef Bacik wrote:
> Hello,
> 
> For the last few years we've been relying on my nightly tests to catch any
> regressions we have in our development tree.  The results have been crudely
> dumped to http://toxicpanda.com and http://toxicpanda.com/performance/.  This is
> a bunch of VM's on a Intel NUC with cronjobs and chewing gum holding it all
> together.  It has worked pretty well, but it is beginning to show it's age, as
> the overnight runs literally take all night, starting around 8PM my time and
> finishing 1:30PM my time.
> 
> I've upgraded my hardware and re-built the VM setup.  The old setup will
> continue so we have a nice baseline of what's going on, but the new setup will
> be triggered with GitHub Actions.
> 
> # tl;dr
> 
> If you are a btrfs developer and are in the https://github.com/btrfs
> organization, you can now submit PR's against the "ci" branch in the "linux"
> repo.  This will trigger a full fstests run using my VM's, which currently test
> 7 different configurations, 5 on x86 and 2 on ARM64.  Zoned will be added
> shortly, I've just been fighting with the thing to get this far so that part is
> still in the pipeline.
> 
> It is highly encouraged that if you've got a decently involved patchset that you
> take advantage of this system to pre-check your code, either before submitting
> it to the list for review or in parallel.

I would recommend to use the CI in addition to any local tests, this
should not be a substitute. All of us have a different configurations of
VMs, machines or .config and this has proven to be very useful. 

> # OMFG GITHUB!?!?
> 
> Yes, I'm nothing if not lazy, and honestly it took me about a week to work out
> all the kinks in this particular system.  We're already on GH, we already have a
> lot of processes around using our existing GH org, so I did this with GitHub
> actions.  I have 0 interest in debating this particular choice.  If you aren't a
> core developer you aren't my target audience.  If you don't want to use the CI
> you don't have to, this is simply a convenience offered to those of us who do
> this for our job.

The GH actions and configuration are partially files tracked in git,
partially doen only on the web (permissions, integration with other
services), it would be good to have it documented somewhere. I put notes
to changelogs when enabling something etc, or we can have a separate
document that enumerates what needs to be clicked and where. I'm not
expecting GH to disappear anytime soon. From past experience the
travis-ci hosted CI for btrfs-progs became unusuable and none of the
scripts could have been used directly but at least it documented which
tests were run, what packages are needed to install etc. Such changes
happen rarely but it is useful to start with something than to spend an
extra week figuring everything out again.