Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run

[Kevin Wolf <kwolf@redhat.com>]

Am 10.05.2019 um 10:55 hat Thomas Huth geschrieben:
> On 08/05/2019 07.47, Thomas Huth wrote:
> > On 07/05/2019 17.50, Eric Blake wrote:
> >> On 5/7/19 10:22 AM, Thomas Huth wrote:
> >>> On 07/05/2019 15.22, Markus Armbruster wrote:
> >>>> Thomas Huth <thuth@redhat.com> writes:
> >>>>
> >>>>> Currently, all tests are in the "auto" group. This is a little bit pointless.
> >>>>> OTOH, we need a group for the tests that we can automatically run during
> >>>>> "make check" each time, too. Tests in this new group are supposed to run
> >>>>> with every possible QEMU configuration, for example they must run with every
> >>>>> QEMU binary (also non-x86), without failing when an optional features is
> >>>>> missing (but reporting "skip" is ok), and be able to run on all kind of host
> >>>>> filesystems and users (i.e. also as "nobody" or "root").
> >>>>> So let's use the "auto" group for this class of tests now. The initial
> >>>>> list has been determined by running the iotests with non-x86 QEMU targets
> >>>>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including
> >>>>> macOS and FreeBSD).
> >>>>
> >>>> I wonder whether we should additionally limit "make check" to "quick"
> >>>> tests.  How slow are the non-quick auto tests for you?
> >>>
> >>> I already sorted out some of the tests that run veeeery long, since the
> >>> run time on gitlab, cirrus-ci and travis is limited. "make check-block"
> >>> currently takes 3 minutes on my laptop, I think that's still ok?
> >>>
> >>> When I run the tests from the auto group that are not in the quick
> >>> group, I currently get:
> >>>
> >>
> >> My personal threshold is about 5 seconds for quick, so:
> >>
> >>> 003 1s ...
> >>> 007 2s ...
> >>
> >> Should these be moved to quick?
> > 
> > I'll leave that decision up to the blocklayer folks ... I thought that
> > there might have been a different reason that these have not been put
> > into "quick" yet...?
> > 
> >>> 013 5s ...
> >>
> >> this one is borderline
> >>
> >>> 014 15s ...
> >>> 015 9s ...
> >>
> >> Definitely not quick, but if you think they are still okay for auto, I
> >> can live with that.
> >>
> >>> 022 1s ...
> >>
> >> Another candidate for quick?
> >>
> >>> 023 18s ...
> >>
> >> Even longer than 14. Okay for auto?
> > 
> > I think I'd give it a try. If people are complaining later that "make
> > check" is running now way too long, we still can refine the list later.
> 
> Thinking about this again, "make check" now runs quite a bit longer
> indeed. So I now rather tend to remove the tests that run longer than 5s
> from the auto group instead... I think I'll send a v4 of this patch
> where I'll remove them from the auto group.

I don't think time is everything. We should also consider how much
the tests contribute to basic code coverage. There is no point in
removing a test from the list because it takes 10 seconds, but if I
split it in two tests taking each 5 seconds, you would include both
halves.

For example, 030, 040 and 041 are not that quick (14/11/42 seconds,
respectively), but they are the most important tests for block jobs and
covering a lot. Sure, 42 seconds is a lot, but I'd keep 030 and 040 at
least.

Kevin