Re: [PATCH 2/3] tests/qemu-iotests: Improve the dry run list to speed up thorough testing

[Kevin Wolf <kwolf@redhat.com>]

Am 10.09.2025 um 17:37 hat Thomas Huth geschrieben:
> From: Thomas Huth <thuth@redhat.com>
> 
> When running the tests in thorough mode, e.g. with:
> 
>  make -j$(nproc) check SPEED=thorough
> 
> we currently always get a huge amount of total tests that the test
> runner tries to execute (2457 in my case), but a big bunch of them are
> only skipped (1099 in my case, meaning that only 1358 got executed).
> This happens because we try to run the whole set of iotests for multiple
> image formats while a lot of the tests can only run with one certain
> format only and thus are marked as SKIP during execution. This is quite a
> waste of time during each test run, and also unnecessarily blows up the
> displayed list of executed tests in the console output.
> 
> Thus let's try to be a little bit smarter: If the "check" script is run
> with "-n" and an image format switch (like "-qed") at the same time (which
> is what we do already for discovering the tests for the meson test runner),
> only report the tests that likely support the given format instead of
> providing the whole list of all tests. We can determine whether a test
> supports a format or not by looking at the lines in the file that contain
> a "supported_fmt" or "unsupported_fmt" statement. This is only heuristics,
> of course, but it is good enough for running the iotests via "make
> check-block" - I double-checked that the list of executed tests does not
> get changed by this patch, it's only the tests that are skipped anyway that
> are now not run anymore.
> 
> This way the amount of total tests drops from 2457 to 1432 for me, and
> the amount of skipped tests drops from 1099 to just 74 (meaning that we
> still properly run 1432 - 74 = 1358 tests as we did before).
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  tests/qemu-iotests/check | 35 ++++++++++++++++++++++++++++++++---
>  1 file changed, 32 insertions(+), 3 deletions(-)

> +def dry_run_list(test_dir, imgfmt, testlist):
> +    for t in testlist:
> +        if not imgfmt:
> +            print('\n'.join([os.path.basename(t)]))
> +            continue
> +        # If a format has been given, we look for the "supported_fmt"
> +        # and the "unsupported_fmt" lines in the test and try to find out
> +        # whether the format is supported or not. This is only heuristics,
> +        # but it should be good enough for "make check-block"

I'm not completely sure if this is a good idea at all, but I think we
should at least mention the possible surprising cases where the
heuristics fails in this comment.

One thing is that '_unsupported_fmt qcow' will also disable the test
case for qcow2. 181 would almost run into this, but because it has
'_supported_fmt generic' first (which is an order not required by
anything else) and you stop after seeing this line, the test as written
currently is still returned (for both qcow and qcow2).

Actually, I suspect that the '_supported_fmt generic' tests that are
then followed by an '_unsupported_fmt' line are the majority of the
still skipped cases you see. (We don't have to do anything about it,
just an observation.)

There are other theoretical scenarios, like multiple options on a single
line in Python, but I think in practice you'll never combine block
driver names that this would skip something it shouldn't skip (apart
from the same 'qcow2' and 'qcow' problem).

Kevin