[LSF/MM/BPF TOPIC] Increasing automation of filesystem testing with kdevops

[Luis Chamberlain <mcgrof@kernel.org>]

Ever since I've taken a dive into filesystems I've been trying to
further automate filesytem setup / testing / collection of results.
I had looked at xfstests-bld [0] but was not happy with it being cloud
specific to Google Compute Engine, and so I have been shopping around
for technology / tooling which would be cloud agnostic / virtualization
agnostic.

At the last LSFMM in Puerto Rico the project oscheck [1] was mentioned a
few times as a mechanism as to how to help get set up fast with fstests,
however *full* automation to include running the tests, processing
results, and updating a baseline was really part of the final plan.
I had not completed the work yet by LSFM Puerto Rico, so could not
talk about the work. The majority of the effort is now complete
and is part of kdevops [2], now a more generic framework to help automated
kernel development testing. I've written a tiny bit about it [3]. Due to
the nature of LSFMM I don't want to present the work, unless folks
really want me to, so would rather have a discussion over technologies
used, pain points to consider, some future ideas, and see what others
are doing. May be worth just as a simple BoF.

So let me start in summary style with some of these on my end.

Technologies used:

  * vagrant / terraform
  * ansible

Pain points:

  * What fstests doesn't cover, or an auto-chinner needed:
    - fsmark regressions, for instance:
      https://lkml.org/lkml/2013/9/10/46
  * vagrant-libvirt is not yet part of upstream vagrant but neeed
    for use with KVM
  * Reliance on only one party (Hashi Corp) for the tooling, even though
    its all open source
  * Vagrant's dependency on ruby and several ruby gems
  * terraform's reliance on tons of go modules
  * "Enterpise Linux" considerations for all the above

Future ideas:

  * Using 9pfs for sharing git trees
  * Does xunit suffice?
  * Evaluating which tests can be folded under kunit
  * Evaluating running one test per container so to fully parallelize testing

[0] https://git.kernel.org/pub/scm/fs/ext2/xfstests-bld.git
[1] https://github.com/mcgrof/oscheck
[2] https://github.com/mcgrof/kdevops
[3] https://people.kernel.org/mcgrof/kdevops-a-devops-framework-for-linux-kernel-development

  Luis