Re: [PATCH 2/3] tests/qemu-iotests: Improve the dry run list to speed up thorough testing

[Kevin Wolf <kwolf@redhat.com>]

Am 02.10.2025 um 16:33 hat Thomas Huth geschrieben:
> On 02/10/2025 16.04, Kevin Wolf wrote:
> > Am 10.09.2025 um 17:37 hat Thomas Huth geschrieben:
> > > From: Thomas Huth <thuth@redhat.com>
> > > 
> > > When running the tests in thorough mode, e.g. with:
> > > 
> > >   make -j$(nproc) check SPEED=thorough
> > > 
> > > we currently always get a huge amount of total tests that the test
> > > runner tries to execute (2457 in my case), but a big bunch of them are
> > > only skipped (1099 in my case, meaning that only 1358 got executed).
> > > This happens because we try to run the whole set of iotests for multiple
> > > image formats while a lot of the tests can only run with one certain
> > > format only and thus are marked as SKIP during execution. This is quite a
> > > waste of time during each test run, and also unnecessarily blows up the
> > > displayed list of executed tests in the console output.
> > > 
> > > Thus let's try to be a little bit smarter: If the "check" script is run
> > > with "-n" and an image format switch (like "-qed") at the same time (which
> > > is what we do already for discovering the tests for the meson test runner),
> > > only report the tests that likely support the given format instead of
> > > providing the whole list of all tests. We can determine whether a test
> > > supports a format or not by looking at the lines in the file that contain
> > > a "supported_fmt" or "unsupported_fmt" statement. This is only heuristics,
> > > of course, but it is good enough for running the iotests via "make
> > > check-block" - I double-checked that the list of executed tests does not
> > > get changed by this patch, it's only the tests that are skipped anyway that
> > > are now not run anymore.
> > > 
> > > This way the amount of total tests drops from 2457 to 1432 for me, and
> > > the amount of skipped tests drops from 1099 to just 74 (meaning that we
> > > still properly run 1432 - 74 = 1358 tests as we did before).
> > > 
> > > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > > ---
> > >   tests/qemu-iotests/check | 35 ++++++++++++++++++++++++++++++++---
> > >   1 file changed, 32 insertions(+), 3 deletions(-)
> > 
> > > +def dry_run_list(test_dir, imgfmt, testlist):
> > > +    for t in testlist:
> > > +        if not imgfmt:
> > > +            print('\n'.join([os.path.basename(t)]))
> > > +            continue
> > > +        # If a format has been given, we look for the "supported_fmt"
> > > +        # and the "unsupported_fmt" lines in the test and try to find out
> > > +        # whether the format is supported or not. This is only heuristics,
> > > +        # but it should be good enough for "make check-block"
> > 
> > I'm not completely sure if this is a good idea at all, but I think we
> > should at least mention the possible surprising cases where the
> > heuristics fails in this comment.
> 
> Yeah, it's not perfect, but I also failed to come up with a better
> solution... do you have any other ideas?

Maybe fixing the qcow vs. qcow2 case could be possible relatively easily
by operating only on full words, but I don't have any fundamentally
different solution without touching every script.

If touching everything is an option, maybe we could have a standardised
comment line like we have with "# group:" today. This would be easy to
parse and work in both bash and Python scripts. But I'm not sure if it's
worth it.

> If not, would you be fine with the patch if I respin it with the comment
> here extended with some words about those surprising cases?

Yes, I just want to have potentially surprising behaviour documented.

Kevin