Re: [Qemu-devel] [PATCH v3 6/7] tests/qemu-iotests/group: Re-use the "auto" group for tests that can always run

[Kevin Wolf <kwolf@redhat.com>]

Am 10.05.2019 um 17:29 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
> 
> > Am 10.05.2019 um 10:55 hat Thomas Huth geschrieben:
> >> On 08/05/2019 07.47, Thomas Huth wrote:
> >> > On 07/05/2019 17.50, Eric Blake wrote:
> >> >> On 5/7/19 10:22 AM, Thomas Huth wrote:
> >> >>> On 07/05/2019 15.22, Markus Armbruster wrote:
> >> >>>> Thomas Huth <thuth@redhat.com> writes:
> >> >>>>
> >> >>>>> Currently, all tests are in the "auto" group. This is a little bit pointless.
> >> >>>>> OTOH, we need a group for the tests that we can automatically run during
> >> >>>>> "make check" each time, too. Tests in this new group are supposed to run
> >> >>>>> with every possible QEMU configuration, for example they must run with every
> >> >>>>> QEMU binary (also non-x86), without failing when an optional features is
> >> >>>>> missing (but reporting "skip" is ok), and be able to run on all kind of host
> >> >>>>> filesystems and users (i.e. also as "nobody" or "root").
> >> >>>>> So let's use the "auto" group for this class of tests now. The initial
> >> >>>>> list has been determined by running the iotests with non-x86 QEMU targets
> >> >>>>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including
> >> >>>>> macOS and FreeBSD).
> >> >>>>
> >> >>>> I wonder whether we should additionally limit "make check" to "quick"
> >> >>>> tests.  How slow are the non-quick auto tests for you?
> >> >>>
> >> >>> I already sorted out some of the tests that run veeeery long, since the
> >> >>> run time on gitlab, cirrus-ci and travis is limited. "make check-block"
> >> >>> currently takes 3 minutes on my laptop, I think that's still ok?
> >> >>>
> >> >>> When I run the tests from the auto group that are not in the quick
> >> >>> group, I currently get:
> >> >>>
> >> >>
> >> >> My personal threshold is about 5 seconds for quick, so:
> >> >>
> >> >>> 003 1s ...
> >> >>> 007 2s ...
> >> >>
> >> >> Should these be moved to quick?
> >> > 
> >> > I'll leave that decision up to the blocklayer folks ... I thought that
> >> > there might have been a different reason that these have not been put
> >> > into "quick" yet...?
> >> > 
> >> >>> 013 5s ...
> >> >>
> >> >> this one is borderline
> >> >>
> >> >>> 014 15s ...
> >> >>> 015 9s ...
> >> >>
> >> >> Definitely not quick, but if you think they are still okay for auto, I
> >> >> can live with that.
> >> >>
> >> >>> 022 1s ...
> >> >>
> >> >> Another candidate for quick?
> >> >>
> >> >>> 023 18s ...
> >> >>
> >> >> Even longer than 14. Okay for auto?
> >> > 
> >> > I think I'd give it a try. If people are complaining later that "make
> >> > check" is running now way too long, we still can refine the list later.
> >> 
> >> Thinking about this again, "make check" now runs quite a bit longer
> >> indeed. So I now rather tend to remove the tests that run longer than 5s
> >> from the auto group instead... I think I'll send a v4 of this patch
> >> where I'll remove them from the auto group.
> >
> > I don't think time is everything. We should also consider how much
> > the tests contribute to basic code coverage. There is no point in
> > removing a test from the list because it takes 10 seconds, but if I
> > split it in two tests taking each 5 seconds, you would include both
> > halves.
> >
> > For example, 030, 040 and 041 are not that quick (14/11/42 seconds,
> > respectively), but they are the most important tests for block jobs and
> > covering a lot. Sure, 42 seconds is a lot, but I'd keep 030 and 040 at
> > least.
> 
> Yes, we want block jobs covered.  However, 42 seconds is a lot indeed.
> Can you think of ways to get a useful part of the full coverage in five
> seconds or less?

I'm not even sure which part of the test is taking so long. Maybe it's
only one of the 88 test functions that's taking long. Or we're
duplicating a lot of tests because we test all combinations of setups
where testing only certain combinations would give enough coverage.

Answering this will probably take some effort checking what the exact
cases tested are and how long each one takes.

Kevin