[LSF/MM/BPF TOPIC] Filesystem testing

[LSF/MM/BPF TOPIC] Filesystem testing

[Leah Rumancik]

Last year we covered the new process for backporting to XFS. There are
still remaining pain points: establishing a baseline for new branches
is time consuming, testing resources aren't easy to come by for
everyone, and selecting appropriate patches is also time consuming. To
avoid the need to establish a baseline, I'm planning on converting to
a model in which I only run failed tests on the baseline. I test with
gce-xfstests and am hoping to automate a relaunch of failed tests.
Perhaps putting the logic to process the results and form new ./check
commands could live in fstests-dev in case it is useful for other
testing infrastructures. As far as patch selection goes, we should
consider what the end goal looks like XFS backporting. One potential
option would be to opt-in for AUTOSEL and for patches that cc stable,
and use bots to automatically check for regressions on queued up
patches - this would require a lot more automation work to be done and
would also depend on the timelines of how long patches are queued
before merged, how long the tests take to run, etc. As for testing
resources, we are still looking for employers to sponsor testing,
ideally in a way that anyone willing to contribute to stable testing
can easily gain access to resources for quick ramp-up.

Re: [LSF/MM/BPF TOPIC] Filesystem testing

[Luis Chamberlain]

On Thu, Mar 14, 2024 at 08:30:28AM -0700, Leah Rumancik wrote:
> Last year we covered the new process for backporting to XFS. There are
> still remaining pain points: establishing a baseline for new branches
> is time consuming,

It's why I think this needs to be automated, the reality is we have
different test runners and we have users of them for different reasons,
and so collaboration on the automation is done within the realms of
each test runner.

We should discuss however things we could share / colaborate on.
I believe the baseline is one of those things we can probably
collaborate on.

We have one for tmpfs and xfs for v6.8-rc2 on kdevops:

https://github.com/linux-kdevops/kdevops/tree/master/workflows/fstests/expunges/6.8.0-rc2

Through dialog at previous LSFMMs we have discussed perhaps sharing
results, those are also shared:

https://github.com/linux-kdevops/kdevops/tree/master/workflows/fstests/results/archive

From a kdevops perspective, the experience is this bloats the tree so
I think this should just be a separate tree, and if you have a path that
exists with it, we copy results to it and push.

Could we share beween different test runners? A directory per each test runner?

I realize that some tests may be private too though, but that does not
preclude us from having public shared results.

> testing resources aren't easy to come by for everyone,

A recurring theme over the last maybe 4-5 LSFMMs.

We have to consider this from those who need or want this. Let's be
frank, enterprise distros already have this well figured out. They've
been doing this for years and they have the experinece and resources
to do it, they just put the focus on their kernels.

That begs the question who needs to be doing serious stable / rc release
testing then?  There are two groups:

  a) kernel filesystem maintainers
  b) consumers of stable kernel

Maintainers can only do what they can, sometimes they don't have enough
time to do it all, as Darrick had mentioned a while ago, it is why the only
way to make things work is to split filesystem maintainer taks into chunks
and hope you can find contributors to help. Otherwise maintainer burn out.
I think the testing requirements for filesystem maintainers then should be
divided and we as a community should help with this effort somehow.

How do we do this? Well you are doing development for a filesystem see
if you company can help with this. We're helping to do some changes on XFS
and tmpfs and so are helping with those. We want to automate that and work
with the community on that. XFS is going to take time and we hope to use
tmpfs as a template for that work.

I recommend we ask the same for other contributors: pick any test runner
for the filesystem you are working on and help lead with the testing.
With either people or hardware. Samsung has helped with both, hardware
and resources. Even if our team is not consumers of stable kernels.  We
help with the latest rc relase baseline for XFS and hope that it is
helpful to XFS stable maintainers. Samsung provides a bare metal server
for interested folks too, even if its not for XFS.  We have asked cloud
providers to help and so far only Oracle has come through on OCI. It
proved *very* important for us as we used it for testing on 64 PAGE_SIZE
systems for some of the testing we did.

So I'd like to ask cloud providers if they can consider allowing
filesystem maintainers to leverage cloud credits to help with testing
filesystems. The testers may not be the maintainer themselves, but
perhaps new developers.

I'd also like to ask if this is a topic that should be discussed with
possible LSFMM sponsors?

Those that fall into camp b) should seriously consider how to help
with the above.

> and selecting appropriate patches is also time consuming.

Although the expectation is that each stable kernel developer
should go through each patch, I hope developers realize that this
is becoming an increasingly difficult task. Sometimes entire complete
areas are re-written and what you knew a few months ago is gone.
So developers working on stable backports need all the help they can
get.

At Linux Plumbers 2023 I asked to help Sasha with automation for the
AUTOSEL setup so other any kernel developer could ramp up with it,
upon review some of the old GPUs needed can be purchased at ebay
today for a raesonable price so that we can get any kernel devloper
who really has the need to help with their own setup.

I'd like to ask again: Sasha, can we get this code up and we document
a way so that any kernel developer can bring up and do their own thing
to help get candidate fixes too?

> To avoid the need to establish a baseline, I'm planning on converting to
> a model in which I only run failed tests on the baseline.

I don't quite follow this. How would you know which tests failed if you
don't have the baseline?

> I test with gce-xfstests and am hoping to automate a relaunch of failed tests.

Kent has previously noted how skipping failed tests is a mistake, I
agree now, but it does not preclude the value in a baseline, or finding
one, or sharing one.

After one has a baseline on can simply re-run test against failed tests always.

> Perhaps putting the logic to process the results and form new ./check
> commands could live in fstests-dev in case it is useful for other
> testing infrastructures.

What would the new check command have as arguments?

> As far as patch selection goes, we should
> consider what the end goal looks like XFS backporting. One potential
> option would be to opt-in for AUTOSEL and for patches that cc stable,
> and use bots to automatically check for regressions on queued up
> patches - this would require a lot more automation work to be done and
> would also depend on the timelines of how long patches are queued
> before merged, how long the tests take to run, etc.

Getting to a sweet spot I think would be to get to the point we have
confidence we can get *some* fstsets patches get tested automatically
tested and results displayed through patchwork.

I hope to get there with tmpfs with kdevops as an initial goal.

> As for testing
> resources, we are still looking for employers to sponsor testing,
> ideally in a way that anyone willing to contribute to stable testing
> can easily gain access to resources for quick ramp-up.

I think we should ask stable consumers for this directly thorugh other
outlets, and other than that, then at LSFMM as contributors to
filesystems to help with this.

  Luis

Re: [LSF/MM/BPF TOPIC] Filesystem testing

[Ritesh Harjani]

Leah Rumancik <leah.rumancik@gmail.com> writes:

> Last year we covered the new process for backporting to XFS. There are
> still remaining pain points: establishing a baseline for new branches
> is time consuming, testing resources aren't easy to come by for
> everyone, and selecting appropriate patches is also time consuming. To
> avoid the need to establish a baseline, I'm planning on converting to
> a model in which I only run failed tests on the baseline. I test with
> gce-xfstests and am hoping to automate a relaunch of failed tests.
> Perhaps putting the logic to process the results and form new ./check
> commands could live in fstests-dev in case it is useful for other
> testing infrastructures.

Nice idea. Another painpoint to add - 
4k blocksize gets tested a lot but as soon as we switch to large block
size testing, either with LBS, or on a system with larger pagesize...
...we quickly starts seeing problems. Most of them could be testcase
failure, so if this could help establish a baseline, that might be helpful.


Also if could collborate on exclude/known failures w.r.t different
test configs that might come handy for people who are looking to help in
this effort. In fact, why not have different filesystems cfg files and their
corresponding exclude files as part of fstests repo itself?  
I know xfstests-bld maintains it here [1][2][3]. And it is rather
very convinient to point this out to anyone who asks me of what test
configs to test with or what tests are considered to be testcase
failures bugs with a given fs config.

So it will very helpful if we could have a mechanism such that all of
this fs configs (and it's correspinding excludes) could be maintained in
fstests itself, and anyone who is looking to test any fs config should
be quickly be able to test it with ./check <fs_cfg_params>. Has this
already been discussed before? Does this sound helpful for people who
are looking to contribute in this effort of fs testing?


[1] [ext4]: https://github.com/tytso/xfstests-bld/tree/master/test-appliance/files/root/fs/ext4/cfg
[2] [xfs]: https://github.com/tytso/xfstests-bld/tree/master/test-appliance/files/root/fs/xfs/cfg
[3] [fs]: https://github.com/tytso/xfstests-bld/tree/master/test-appliance/files/root/fs/

-ritesh

Re: [LSF/MM/BPF TOPIC] Filesystem testing

[Gabriel Krisman Bertazi]

Ritesh Harjani (IBM) <ritesh.list@gmail.com> writes:

> Leah Rumancik <leah.rumancik@gmail.com> writes:
>
>> Last year we covered the new process for backporting to XFS. There are
>> still remaining pain points: establishing a baseline for new branches
>> is time consuming, testing resources aren't easy to come by for
>> everyone, and selecting appropriate patches is also time consuming. To
>> avoid the need to establish a baseline, I'm planning on converting to
>> a model in which I only run failed tests on the baseline. I test with
>> gce-xfstests and am hoping to automate a relaunch of failed tests.
>> Perhaps putting the logic to process the results and form new ./check
>> commands could live in fstests-dev in case it is useful for other
>> testing infrastructures.
>
> Nice idea. Another painpoint to add - 
> 4k blocksize gets tested a lot but as soon as we switch to large block
> size testing, either with LBS, or on a system with larger pagesize...
> ...we quickly starts seeing problems. Most of them could be testcase
> failure, so if this could help establish a baseline, that might be helpful.
>
>
> Also if could collborate on exclude/known failures w.r.t different
> test configs that might come handy for people who are looking to help in
> this effort. In fact, why not have different filesystems cfg files and their
> corresponding exclude files as part of fstests repo itself?  
> I know xfstests-bld maintains it here [1][2][3]. And it is rather
> very convinient to point this out to anyone who asks me of what test
> configs to test with or what tests are considered to be testcase
> failures bugs with a given fs config.
>
> So it will very helpful if we could have a mechanism such that all of
> this fs configs (and it's correspinding excludes) could be maintained in
> fstests itself, and anyone who is looking to test any fs config should
> be quickly be able to test it with ./check <fs_cfg_params>. Has this
> already been discussed before? Does this sound helpful for people who
> are looking to contribute in this effort of fs testing?
>
>
> [1] [ext4]:
> https://github.com/tytso/xfstests-bld/tree/master/test-appliance/files/root/fs/ext4/cfg

Looking at the expunge comments, I think many of those entries should
just be turned into inline checks in the test preamble and skipped with
_notrun.  The way I see it, expunged tests should be kept to a minimum,
and the goal should be to eventually remove them from the list, IMO.
They are tests that are known to be broken or flaky now, and can be safely
ignored when doing unrelated work, but that will be fixed in the
future. Tests that will always fail because the feature doesn't exist in
the filesystem, or because it asks for an impossible situation in a
specific configuration should be checked inline and skipped, IMO.

+1 for the idea of having this in fstests.  Even if we
lack the infrastructure to do anything useful with it in ./check,
having them in fstests will improve collaboration throughout
different fstests wrappers (kernelci, xfstests-bld, etc.)

> [2] [xfs]: https://github.com/tytso/xfstests-bld/tree/master/test-appliance/files/root/fs/xfs/cfg
> [3] [fs]: https://github.com/tytso/xfstests-bld/tree/master/test-appliance/files/root/fs/



>
> -ritesh

-- 
Gabriel Krisman Bertazi

Re: [LSF/MM/BPF TOPIC] Filesystem testing

[Dave Chinner]

On Mon, Mar 18, 2024 at 02:48:51PM -0400, Gabriel Krisman Bertazi wrote:
> Ritesh Harjani (IBM) <ritesh.list@gmail.com> writes:
> 
> > Leah Rumancik <leah.rumancik@gmail.com> writes:
> >
> >> Last year we covered the new process for backporting to XFS. There are
> >> still remaining pain points: establishing a baseline for new branches
> >> is time consuming, testing resources aren't easy to come by for
> >> everyone, and selecting appropriate patches is also time consuming. To
> >> avoid the need to establish a baseline, I'm planning on converting to
> >> a model in which I only run failed tests on the baseline. I test with
> >> gce-xfstests and am hoping to automate a relaunch of failed tests.
> >> Perhaps putting the logic to process the results and form new ./check
> >> commands could live in fstests-dev in case it is useful for other
> >> testing infrastructures.
> >
> > Nice idea. Another painpoint to add - 
> > 4k blocksize gets tested a lot but as soon as we switch to large block
> > size testing, either with LBS, or on a system with larger pagesize...
> > ...we quickly starts seeing problems. Most of them could be testcase
> > failure, so if this could help establish a baseline, that might be helpful.
> >
> >
> > Also if could collborate on exclude/known failures w.r.t different
> > test configs that might come handy for people who are looking to help in
> > this effort. In fact, why not have different filesystems cfg files and their
> > corresponding exclude files as part of fstests repo itself?  
> > I know xfstests-bld maintains it here [1][2][3]. And it is rather
> > very convinient to point this out to anyone who asks me of what test
> > configs to test with or what tests are considered to be testcase
> > failures bugs with a given fs config.
> >
> > So it will very helpful if we could have a mechanism such that all of
> > this fs configs (and it's correspinding excludes) could be maintained in
> > fstests itself, and anyone who is looking to test any fs config should
> > be quickly be able to test it with ./check <fs_cfg_params>. Has this
> > already been discussed before? Does this sound helpful for people who
> > are looking to contribute in this effort of fs testing?

Filesystem configs have already been implemented, yes? i.e. config
file sections.

We can do delta definitions like this in the config file:

RECREATE_TEST_DEV=true
TEST_MNT=/mnt/test
TEST_DEV=/dev/vda
SCRATCH_MNT=/mnt/scratch
SCRATCH_DEV=/dev/vdb
MKFS_OPTIONS=
MOUNT_OPTIONS=

[xfs_4k]
MKFS_OPTIONS="-m rmapbt=1"

[xfs_4k_quota]
MKFS_OPTIONS="-m rmapbt=1"
MOUNT_OPTIONS="-o uquota,gquota,pquota"

[xfs_1k]
MKFS_OPTIONS="-m rmapbt=1 -b size=1k"
MOUNT_OPTIONS=

[xfs_n64k]
MKFS_OPTIONS="-m rmapbt=1 -n size=64k"

....

And then simply run 'check -s xfs_n64k' or "-s xfs_4k_quota" or
"-s xfs_1k", etc to run the tests against a pre-defined filesystem
configuration.

The actual per-system customised part of the config file is the
initial device and mount definitions, all the fs config definitions
are fixed and never really change. So we could ship a config file
like the above as a template alongside config/example.config (e.g.
example.xfs.config) and then the test environment setup can simply
copy the file and use sed to rewrite the devices/mount points to
match what it is going to use...

IOWs, I think the fs config thing is already a solved problem, and
we already have precedent for shipping example config files...

As for excludes - unlike fs configs, these are not static across all
test environments. They are entirely dependent on what
kernel/userspace combination is being tested and the constraints the
test running is executing under (e.g.  runtime constraints). IOWs,
every external test runner has a different set of tests that it will
need to expunge...

As it is, it would be trivial to add a config file section variable
to define an expunge file for a given config section. That way
the test running could keep it's own expunge files and add them
to the relevant section when setting up the test VM environment,
same as it would do for the devices and mounts.

That way the expunge file isn't needed on the CLI, and so the test
runner could just do 'check -s xfs_4k -s xfs_1k -s xfs_4k_quota" and
get all those configs tested and have all the local expunges for the
different configs just work....

> > [1] [ext4]:
> > https://github.com/tytso/xfstests-bld/tree/master/test-appliance/files/root/fs/ext4/cfg
> 
> Looking at the expunge comments, I think many of those entries should
> just be turned into inline checks in the test preamble and skipped with
> _notrun.

This is the right thing to do - reduce the reliance on expunge
files, and hence get rid of the need for them in most cases
altogether. The best code is -no code-.

> The way I see it, expunged tests should be kept to a minimum,
> and the goal should be to eventually remove them from the list, IMO.
> They are tests that are known to be broken or flaky now, and can be safely
> ignored when doing unrelated work, but that will be fixed in the
> future. Tests that will always fail because the feature doesn't exist in
> the filesystem, or because it asks for an impossible situation in a
> specific configuration should be checked inline and skipped, IMO.

> +1 for the idea of having this in fstests.  Even if we
> lack the infrastructure to do anything useful with it in ./check,
> having them in fstests will improve collaboration throughout
> different fstests wrappers (kernelci, xfstests-bld, etc.)

Except that this places the maintenance burden on fstests, in
an environment where we can do -nothing- to validate the correctness
of these lists, nor have any idea of when tests should or
shouldn't be placed in these lists.

i.e. If your test runner needs to expunge tests for some reason,
either keep the expunge lists with the test runner, or add detection
to the test that automatically _notrun()s the test in enviroments
where it shouldn't be run....

I'd much prefer the improvement of _notrun detection over spreading
the expunge file mess further into fstests. THis helps remove the
technical debt (lack of proper checking in the test) rather than
kicking it down the road for someone else to have to deal with in
future.

Centralisation of third party expunge file management is not the
answer.  We should be trying to reduce our reliance on expunges and
the maintenance overhead they require, not driving that expunge file
maintaintenance overhead into fstests itself...

-Dave.

-- 
Dave Chinner
david@fromorbit.com

Re: [LSF/MM/BPF TOPIC] Filesystem testing

[Luis Chamberlain]

On Tue, Mar 19, 2024 at 09:06:18AM +1100, Dave Chinner wrote:
> On Mon, Mar 18, 2024 at 02:48:51PM -0400, Gabriel Krisman Bertazi wrote:
> > +1 for the idea of having this in fstests.  Even if we
> > lack the infrastructure to do anything useful with it in ./check,
> > having them in fstests will improve collaboration throughout
> > different fstests wrappers (kernelci, xfstests-bld, etc.)
> 
> Except that this places the maintenance burden on fstests, in
> an environment where we can do -nothing- to validate the correctness
> of these lists, nor have any idea of when tests should or
> shouldn't be placed in these lists.
> 
> i.e. If your test runner needs to expunge tests for some reason,
> either keep the expunge lists with the test runner, or add detection
> to the test that automatically _notrun()s the test in enviroments
> where it shouldn't be run....
> 
> I'd much prefer the improvement of _notrun detection over spreading
> the expunge file mess further into fstests. THis helps remove the
> technical debt (lack of proper checking in the test) rather than
> kicking it down the road for someone else to have to deal with in
> future.
> 
> Centralisation of third party expunge file management is not the
> answer.  We should be trying to reduce our reliance on expunges and
> the maintenance overhead they require, not driving that expunge file
> maintaintenance overhead into fstests itself...

kdevops has been using expunges since day 1 and shared them. We have one
per filesystem test section and parallelize each test section. While
useful for a baseline, over time I have to agree that a desirable goal
is to not rely on them. But that just means your test runner can deal
with crashes automatically. That is work we've been doing for kdevops
and hope to get there.

That does not preclude the value of a baseline for a kernel too though
test section section. While I agree that it will depend on your
version of fstests and userspace too, its worthwile asking if a generic
kernel baseline is desirable. The answer to this really is about scaling
and doing the work.

An example of a baseline of known critical failures for v6.6:

https://github.com/linux-kdevops/kdevops/blob/main/docs/xfs-bugs.md

Is something like this useful? If so, should we collaborate on a central
one? How?

   Luis