test not in the auto group, was: Re: [PATCH 23/34] btrfs: allow btrfs_submit_bio to split bios

test not in the auto group, was: Re: [PATCH 23/34] btrfs: allow btrfs_submit_bio to split bios

[Christoph Hellwig]

On Thu, Jan 26, 2023 at 07:33:04PM +0100, David Sterba wrote:
> > Oh, I guess the lack of auto group means I've never tested it.  But
> > it's a fairly bad bug, and I'm surprised nothing in auto hits an
> > error after a bio split.  I'll need to find out if I can find a simpler
> > reproducer as this warrants a regression test.
> 
> The 'auto' group is good for first tests, I'm running 'check -g all' on
> my VM setups. If this is enough to trigger errors then we probably don't
> need a separate regression test.

Hmm.  The xfstests README says:

"By default the tests suite will run all the tests in the auto group. These
 are the tests that are expected to function correctly as regression tests,
 and it excludes tests that exercise conditions known to cause machine
 failures (i.e. the "dangerous" tests)."

and my assumptions over decades of xfstests use has been that only tests
that are broken, non-deterministic, or cause recent upstream kernels
to crash are not in auto.

Is there some kind of different rule for btrfs?  e.g. btrfs/125
seems to complete quickly and does not actually seem to be dangerous.
Besides that there's btrfs/185, which is very quick fuzzer, and btrfs/198
which is a fairly normal test as far as I can tell.

The generic tests also have a few !auto tests that look like they
should be mostly in the auto group as well, in addition to a few broken
and dangerous ones, and the blockdev ones from Darrick that should
probably move to blktests.

XFS mostly seems to have dangerous fuzzer tests in the !auto category.