xfstests-bld kernel config and automated tsting (Was: blktests: status, expansion plan for the storage stack test framework)

["Theodore Tso" <tytso@mit.edu>]

On Mon, May 11, 2026 at 09:27:38PM +0900, Shin'ichiro Kawasaki wrote:
> 
> - For easier config:
>   - Ted has the config generation tool.
>   -> I'm interested in it, and would like to take a look.

The config generation tool is part of my xfstests-bld test appliance:

     https://github.com/tytso/xfstests-bld

I have blktests support integrated into gce-xfstests.  After setting
gce-xfstests per:

https://github.com/tytso/xfstests-bld/blob/master/Documentation/gce-xfstests.md

Using it is basically as simple as:

cd /usr/src/linux
install-kconfig --blktests
kbuild
gce-xfstests --blktests

The install-kconfig and kbuild scripts can be found in the
kernel-build directory in the xfstests-bld repo:

https://github.com/tytso/xfstests-bld/tree/master/kernel-build

Note: In addition to --blktests, install-kconfig supports other options,
including --lockdep, --kasan, --ubsan, --arch arm64, --kcsan, etc.


Unfortunately, blktests is currently not supported in kvm-xfstests
(aka qemu-xfstests, since it also works on MacOS's HVF).  It should be
possible, but currently kvm-xfstests doesn't suport modules, and
blktests is pretty fundamentally tied to modules.  With gce-xfstests,
we upload a kernel.deb install it in the system under test.  With
kvm-xfstests, we directly boot the kernel directly out of the build
tree, which is much faster (5 seconds from "kvm-xfstests shell" to
shell prompt)--- but doesn't support modules.


Projects to improve blktests in my test appliance:

   1. Add support for modules in kvm-xfstests and then enable blktests for
      kvm-xfstests.
   2. Add support for some kind of machine readable test results file
      (e.g., junit XML or  kTAP)

The reason why (2) is so interesting is that it allows for a
convenient report generation which summarizes the test run, e.g:

ext4/4k: 615 tests, 1 failures, 74 skipped, 4701 seconds
  Flaky: generic/234: 20% (1/5)
ext4/1k: 609 tests, 78 skipped, 5586 seconds
ext4/ext3: 607 tests, 159 skipped, 5044 seconds
ext4/encrypt: 590 tests, 191 skipped, 2768 seconds
ext4/nojournal: 607 tests, 1 failures, 158 skipped, 4111 seconds
  Flaky: generic/234: 20% (1/5)
ext4/ext3conv: 612 tests, 2 failures, 76 skipped, 5212 seconds
  Flaky: generic/234: 20% (1/5)   generic/409: 20% (1/5)
     
(Note: ext4/4k, ext4/1k, etc. are different file system test
scenarios, which run on a different VM)

The other reason why (2) is handy is that it enables using automated
bisection searches, e.g.:

   gce-xfstests ltm --repo linux-stable-gc -c ext4/4k generic/234 \
       --bisect-good v7.0.5 --bisect-bad linux-7.0.y

So if we had a machine-readable results file, ala xfstests's "check -R
xunit-quiet", I could add automated bisection for blktests failures to
gce-xfstests.

Cheers,

					- Ted

P.S.  For more information about my test-appliance system, see:

https://thunk.org/gce-xfstests
https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md
https://github.com/tytso/xfstests-bld/blob/master/Documentation/gce-xfstests.md