Re: [PATCH 2/3] tests/qemu-iotests: Improve the dry run list to speed up thorough testing

[Thomas Huth <thuth@redhat.com>]

On 02/10/2025 16.04, Kevin Wolf wrote:
> Am 10.09.2025 um 17:37 hat Thomas Huth geschrieben:
>> From: Thomas Huth <thuth@redhat.com>
>>
>> When running the tests in thorough mode, e.g. with:
>>
>>   make -j$(nproc) check SPEED=thorough
>>
>> we currently always get a huge amount of total tests that the test
>> runner tries to execute (2457 in my case), but a big bunch of them are
>> only skipped (1099 in my case, meaning that only 1358 got executed).
>> This happens because we try to run the whole set of iotests for multiple
>> image formats while a lot of the tests can only run with one certain
>> format only and thus are marked as SKIP during execution. This is quite a
>> waste of time during each test run, and also unnecessarily blows up the
>> displayed list of executed tests in the console output.
>>
>> Thus let's try to be a little bit smarter: If the "check" script is run
>> with "-n" and an image format switch (like "-qed") at the same time (which
>> is what we do already for discovering the tests for the meson test runner),
>> only report the tests that likely support the given format instead of
>> providing the whole list of all tests. We can determine whether a test
>> supports a format or not by looking at the lines in the file that contain
>> a "supported_fmt" or "unsupported_fmt" statement. This is only heuristics,
>> of course, but it is good enough for running the iotests via "make
>> check-block" - I double-checked that the list of executed tests does not
>> get changed by this patch, it's only the tests that are skipped anyway that
>> are now not run anymore.
>>
>> This way the amount of total tests drops from 2457 to 1432 for me, and
>> the amount of skipped tests drops from 1099 to just 74 (meaning that we
>> still properly run 1432 - 74 = 1358 tests as we did before).
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   tests/qemu-iotests/check | 35 ++++++++++++++++++++++++++++++++---
>>   1 file changed, 32 insertions(+), 3 deletions(-)
> 
>> +def dry_run_list(test_dir, imgfmt, testlist):
>> +    for t in testlist:
>> +        if not imgfmt:
>> +            print('\n'.join([os.path.basename(t)]))
>> +            continue
>> +        # If a format has been given, we look for the "supported_fmt"
>> +        # and the "unsupported_fmt" lines in the test and try to find out
>> +        # whether the format is supported or not. This is only heuristics,
>> +        # but it should be good enough for "make check-block"
> 
> I'm not completely sure if this is a good idea at all, but I think we
> should at least mention the possible surprising cases where the
> heuristics fails in this comment.

Yeah, it's not perfect, but I also failed to come up with a better 
solution... do you have any other ideas?

If not, would you be fine with the patch if I respin it with the comment 
here extended with some words about those surprising cases?

  Thomas