[PATCH 0/3] tests/qemu-iotests: Improve test detection for meson

[PATCH 0/3] tests/qemu-iotests: Improve test detection for meson

[Thomas Huth]

When running the test in thorough mode, we currently try to run a lot of
tests that are only skipped, which is somewhat annoying. This series
fixes the problems (see second patch), and add some more image formats
in the third patch (which should be fine now that the list of tests that
we try to execute is not that big anymore).

Thomas Huth (3):
  tests/qemu-iotests/184: Fix skip message for qemu-img without throttle
  tests/qemu-iotests: Improve the dry run list to speed up thorough
    testing
  tests/qemu-iotest: Add more image formats to the thorough testing

 tests/qemu-iotests/184         |  2 +-
 tests/qemu-iotests/check       | 35 +++++++++++++++++++++++++++++++---
 tests/qemu-iotests/meson.build |  3 +++
 3 files changed, 36 insertions(+), 4 deletions(-)

-- 
2.51.0

[PATCH 1/3] tests/qemu-iotests/184: Fix skip message for qemu-img without throttle

[Thomas Huth]

From: Thomas Huth <thuth@redhat.com>

If qemu-img does not support throttling, test 184 currently skips
with the message:

  not suitable for this image format: raw

But that's wrong, it's not about the image format, it's about the
throttling not being available in qemu-img. Thus fix this by using
_notrun with a proper message instead.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/qemu-iotests/184 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184
index e4cbcd86345..f7fcedacfb5 100755
--- a/tests/qemu-iotests/184
+++ b/tests/qemu-iotests/184
@@ -50,7 +50,7 @@ run_qemu()
 }
 
 test_throttle=$($QEMU_IMG --help|grep throttle)
-[ "$test_throttle" = "" ] && _supported_fmt throttle
+[ "$test_throttle" = "" ] && _notrun "qemu-img does not support throttle"
 
 echo
 echo "== checking interface =="
-- 
2.51.0

[PATCH 2/3] tests/qemu-iotests: Improve the dry run list to speed up thorough testing

[Thomas Huth]

From: Thomas Huth <thuth@redhat.com>

When running the tests in thorough mode, e.g. with:

 make -j$(nproc) check SPEED=thorough

we currently always get a huge amount of total tests that the test
runner tries to execute (2457 in my case), but a big bunch of them are
only skipped (1099 in my case, meaning that only 1358 got executed).
This happens because we try to run the whole set of iotests for multiple
image formats while a lot of the tests can only run with one certain
format only and thus are marked as SKIP during execution. This is quite a
waste of time during each test run, and also unnecessarily blows up the
displayed list of executed tests in the console output.

Thus let's try to be a little bit smarter: If the "check" script is run
with "-n" and an image format switch (like "-qed") at the same time (which
is what we do already for discovering the tests for the meson test runner),
only report the tests that likely support the given format instead of
providing the whole list of all tests. We can determine whether a test
supports a format or not by looking at the lines in the file that contain
a "supported_fmt" or "unsupported_fmt" statement. This is only heuristics,
of course, but it is good enough for running the iotests via "make
check-block" - I double-checked that the list of executed tests does not
get changed by this patch, it's only the tests that are skipped anyway that
are now not run anymore.

This way the amount of total tests drops from 2457 to 1432 for me, and
the amount of skipped tests drops from 1099 to just 74 (meaning that we
still properly run 1432 - 74 = 1358 tests as we did before).

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/qemu-iotests/check | 35 ++++++++++++++++++++++++++++++++---
 1 file changed, 32 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index 545f9ec7bdd..fb3d8f95b94 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -81,7 +81,7 @@ def make_argparser() -> argparse.ArgumentParser:
     g_env.add_argument('-i', dest='aiomode', default='threads',
                        help='sets AIOMODE environment variable')
 
-    p.set_defaults(imgfmt='raw', imgproto='file')
+    p.set_defaults(imgproto='file')
 
     format_list = ['raw', 'bochs', 'cloop', 'parallels', 'qcow', 'qcow2',
                    'qed', 'vdi', 'vpc', 'vhdx', 'vmdk', 'luks', 'dmg', 'vvfat']
@@ -136,12 +136,41 @@ def make_argparser() -> argparse.ArgumentParser:
     return p
 
 
+def dry_run_list(test_dir, imgfmt, testlist):
+    for t in testlist:
+        if not imgfmt:
+            print('\n'.join([os.path.basename(t)]))
+            continue
+        # If a format has been given, we look for the "supported_fmt"
+        # and the "unsupported_fmt" lines in the test and try to find out
+        # whether the format is supported or not. This is only heuristics,
+        # but it should be good enough for "make check-block"
+        with open(os.path.join(test_dir, t), 'r', encoding='utf-8') as fh:
+            supported = True
+            check_next_line = False
+            for line in fh:
+                if 'unsupported_fmt' in line:
+                    if imgfmt in line:
+                        supported = False
+                        break
+                elif 'supported_fmt' in line or check_next_line:
+                    supported = imgfmt in line or 'generic' in line
+                    check_next_line = not ']' in line and \
+                            ('supported_fmts=[' in line or check_next_line)
+                    if supported or not check_next_line:
+                        break
+            if supported:
+                print('\n'.join([os.path.basename(t)]))
+
+
 if __name__ == '__main__':
     args = make_argparser().parse_args()
 
+    image_format = args.imgfmt or 'raw'
+
     env = TestEnv(source_dir=args.source_dir,
                   build_dir=args.build_dir,
-                  imgfmt=args.imgfmt, imgproto=args.imgproto,
+                  imgfmt=image_format, imgproto=args.imgproto,
                   aiomode=args.aiomode, cachemode=args.cachemode,
                   imgopts=args.imgopts, misalign=args.misalign,
                   debug=args.debug, valgrind=args.valgrind,
@@ -185,7 +214,7 @@ if __name__ == '__main__':
 
     if args.dry_run:
         with env:
-            print('\n'.join([os.path.basename(t) for t in tests]))
+            dry_run_list(env.source_iotests, args.imgfmt, tests)
     else:
         with TestRunner(env, tap=args.tap,
                         color=args.color) as tr:
-- 
2.51.0

[PATCH 3/3] tests/qemu-iotest: Add more image formats to the thorough testing

[Thomas Huth]

From: Thomas Huth <thuth@redhat.com>

Now that the "check" script is a little bit smarter with providing
a list of tests that are supported for an image format, we can also
add more image formats that can be used for generic block layer
testing. (Note: qcow1 and luks are not added because some tests
there currently fail, and other formats like bochs, cloop, dmg and
vvfat do not work with the generic tests and thus would only get
skipped if we'd tried to add them here)

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/qemu-iotests/meson.build | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
index fad340ad595..35256217a37 100644
--- a/tests/qemu-iotests/meson.build
+++ b/tests/qemu-iotests/meson.build
@@ -21,7 +21,10 @@ qemu_iotests_env = {'PYTHON': python.full_path()}
 qemu_iotests_formats = {
   'qcow2': 'quick',
   'raw': 'slow',
+  'parallels': 'thorough',
   'qed': 'thorough',
+  'vdi': 'thorough',
+  'vhdx': 'thorough',
   'vmdk': 'thorough',
   'vpc': 'thorough'
 }
-- 
2.51.0

Re: [PATCH 2/3] tests/qemu-iotests: Improve the dry run list to speed up thorough testing

[Kevin Wolf]

Am 10.09.2025 um 17:37 hat Thomas Huth geschrieben:
> From: Thomas Huth <thuth@redhat.com>
> 
> When running the tests in thorough mode, e.g. with:
> 
>  make -j$(nproc) check SPEED=thorough
> 
> we currently always get a huge amount of total tests that the test
> runner tries to execute (2457 in my case), but a big bunch of them are
> only skipped (1099 in my case, meaning that only 1358 got executed).
> This happens because we try to run the whole set of iotests for multiple
> image formats while a lot of the tests can only run with one certain
> format only and thus are marked as SKIP during execution. This is quite a
> waste of time during each test run, and also unnecessarily blows up the
> displayed list of executed tests in the console output.
> 
> Thus let's try to be a little bit smarter: If the "check" script is run
> with "-n" and an image format switch (like "-qed") at the same time (which
> is what we do already for discovering the tests for the meson test runner),
> only report the tests that likely support the given format instead of
> providing the whole list of all tests. We can determine whether a test
> supports a format or not by looking at the lines in the file that contain
> a "supported_fmt" or "unsupported_fmt" statement. This is only heuristics,
> of course, but it is good enough for running the iotests via "make
> check-block" - I double-checked that the list of executed tests does not
> get changed by this patch, it's only the tests that are skipped anyway that
> are now not run anymore.
> 
> This way the amount of total tests drops from 2457 to 1432 for me, and
> the amount of skipped tests drops from 1099 to just 74 (meaning that we
> still properly run 1432 - 74 = 1358 tests as we did before).
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  tests/qemu-iotests/check | 35 ++++++++++++++++++++++++++++++++---
>  1 file changed, 32 insertions(+), 3 deletions(-)

> +def dry_run_list(test_dir, imgfmt, testlist):
> +    for t in testlist:
> +        if not imgfmt:
> +            print('\n'.join([os.path.basename(t)]))
> +            continue
> +        # If a format has been given, we look for the "supported_fmt"
> +        # and the "unsupported_fmt" lines in the test and try to find out
> +        # whether the format is supported or not. This is only heuristics,
> +        # but it should be good enough for "make check-block"

I'm not completely sure if this is a good idea at all, but I think we
should at least mention the possible surprising cases where the
heuristics fails in this comment.

One thing is that '_unsupported_fmt qcow' will also disable the test
case for qcow2. 181 would almost run into this, but because it has
'_supported_fmt generic' first (which is an order not required by
anything else) and you stop after seeing this line, the test as written
currently is still returned (for both qcow and qcow2).

Actually, I suspect that the '_supported_fmt generic' tests that are
then followed by an '_unsupported_fmt' line are the majority of the
still skipped cases you see. (We don't have to do anything about it,
just an observation.)

There are other theoretical scenarios, like multiple options on a single
line in Python, but I think in practice you'll never combine block
driver names that this would skip something it shouldn't skip (apart
from the same 'qcow2' and 'qcow' problem).

Kevin

Re: [PATCH 1/3] tests/qemu-iotests/184: Fix skip message for qemu-img without throttle

[Kevin Wolf]

Am 10.09.2025 um 17:37 hat Thomas Huth geschrieben:
> From: Thomas Huth <thuth@redhat.com>
> 
> If qemu-img does not support throttling, test 184 currently skips
> with the message:
> 
>   not suitable for this image format: raw
> 
> But that's wrong, it's not about the image format, it's about the
> throttling not being available in qemu-img. Thus fix this by using
> _notrun with a proper message instead.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>

Reviewed-by: Kevin Wolf <kwolf@redhat.com>

Re: [PATCH 3/3] tests/qemu-iotest: Add more image formats to the thorough testing

[Kevin Wolf]

Am 10.09.2025 um 17:37 hat Thomas Huth geschrieben:
> From: Thomas Huth <thuth@redhat.com>
> 
> Now that the "check" script is a little bit smarter with providing
> a list of tests that are supported for an image format, we can also
> add more image formats that can be used for generic block layer
> testing. (Note: qcow1 and luks are not added because some tests
> there currently fail, and other formats like bochs, cloop, dmg and
> vvfat do not work with the generic tests and thus would only get
> skipped if we'd tried to add them here)
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>

Reviewed-by: Kevin Wolf <kwolf@redhat.com>

Re: [PATCH 2/3] tests/qemu-iotests: Improve the dry run list to speed up thorough testing

[Thomas Huth]

On 02/10/2025 16.04, Kevin Wolf wrote:
> Am 10.09.2025 um 17:37 hat Thomas Huth geschrieben:
>> From: Thomas Huth <thuth@redhat.com>
>>
>> When running the tests in thorough mode, e.g. with:
>>
>>   make -j$(nproc) check SPEED=thorough
>>
>> we currently always get a huge amount of total tests that the test
>> runner tries to execute (2457 in my case), but a big bunch of them are
>> only skipped (1099 in my case, meaning that only 1358 got executed).
>> This happens because we try to run the whole set of iotests for multiple
>> image formats while a lot of the tests can only run with one certain
>> format only and thus are marked as SKIP during execution. This is quite a
>> waste of time during each test run, and also unnecessarily blows up the
>> displayed list of executed tests in the console output.
>>
>> Thus let's try to be a little bit smarter: If the "check" script is run
>> with "-n" and an image format switch (like "-qed") at the same time (which
>> is what we do already for discovering the tests for the meson test runner),
>> only report the tests that likely support the given format instead of
>> providing the whole list of all tests. We can determine whether a test
>> supports a format or not by looking at the lines in the file that contain
>> a "supported_fmt" or "unsupported_fmt" statement. This is only heuristics,
>> of course, but it is good enough for running the iotests via "make
>> check-block" - I double-checked that the list of executed tests does not
>> get changed by this patch, it's only the tests that are skipped anyway that
>> are now not run anymore.
>>
>> This way the amount of total tests drops from 2457 to 1432 for me, and
>> the amount of skipped tests drops from 1099 to just 74 (meaning that we
>> still properly run 1432 - 74 = 1358 tests as we did before).
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   tests/qemu-iotests/check | 35 ++++++++++++++++++++++++++++++++---
>>   1 file changed, 32 insertions(+), 3 deletions(-)
> 
>> +def dry_run_list(test_dir, imgfmt, testlist):
>> +    for t in testlist:
>> +        if not imgfmt:
>> +            print('\n'.join([os.path.basename(t)]))
>> +            continue
>> +        # If a format has been given, we look for the "supported_fmt"
>> +        # and the "unsupported_fmt" lines in the test and try to find out
>> +        # whether the format is supported or not. This is only heuristics,
>> +        # but it should be good enough for "make check-block"
> 
> I'm not completely sure if this is a good idea at all, but I think we
> should at least mention the possible surprising cases where the
> heuristics fails in this comment.

Yeah, it's not perfect, but I also failed to come up with a better 
solution... do you have any other ideas?

If not, would you be fine with the patch if I respin it with the comment 
here extended with some words about those surprising cases?

  Thomas

Re: [PATCH 2/3] tests/qemu-iotests: Improve the dry run list to speed up thorough testing

[Kevin Wolf]

Am 02.10.2025 um 16:33 hat Thomas Huth geschrieben:
> On 02/10/2025 16.04, Kevin Wolf wrote:
> > Am 10.09.2025 um 17:37 hat Thomas Huth geschrieben:
> > > From: Thomas Huth <thuth@redhat.com>
> > > 
> > > When running the tests in thorough mode, e.g. with:
> > > 
> > >   make -j$(nproc) check SPEED=thorough
> > > 
> > > we currently always get a huge amount of total tests that the test
> > > runner tries to execute (2457 in my case), but a big bunch of them are
> > > only skipped (1099 in my case, meaning that only 1358 got executed).
> > > This happens because we try to run the whole set of iotests for multiple
> > > image formats while a lot of the tests can only run with one certain
> > > format only and thus are marked as SKIP during execution. This is quite a
> > > waste of time during each test run, and also unnecessarily blows up the
> > > displayed list of executed tests in the console output.
> > > 
> > > Thus let's try to be a little bit smarter: If the "check" script is run
> > > with "-n" and an image format switch (like "-qed") at the same time (which
> > > is what we do already for discovering the tests for the meson test runner),
> > > only report the tests that likely support the given format instead of
> > > providing the whole list of all tests. We can determine whether a test
> > > supports a format or not by looking at the lines in the file that contain
> > > a "supported_fmt" or "unsupported_fmt" statement. This is only heuristics,
> > > of course, but it is good enough for running the iotests via "make
> > > check-block" - I double-checked that the list of executed tests does not
> > > get changed by this patch, it's only the tests that are skipped anyway that
> > > are now not run anymore.
> > > 
> > > This way the amount of total tests drops from 2457 to 1432 for me, and
> > > the amount of skipped tests drops from 1099 to just 74 (meaning that we
> > > still properly run 1432 - 74 = 1358 tests as we did before).
> > > 
> > > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > > ---
> > >   tests/qemu-iotests/check | 35 ++++++++++++++++++++++++++++++++---
> > >   1 file changed, 32 insertions(+), 3 deletions(-)
> > 
> > > +def dry_run_list(test_dir, imgfmt, testlist):
> > > +    for t in testlist:
> > > +        if not imgfmt:
> > > +            print('\n'.join([os.path.basename(t)]))
> > > +            continue
> > > +        # If a format has been given, we look for the "supported_fmt"
> > > +        # and the "unsupported_fmt" lines in the test and try to find out
> > > +        # whether the format is supported or not. This is only heuristics,
> > > +        # but it should be good enough for "make check-block"
> > 
> > I'm not completely sure if this is a good idea at all, but I think we
> > should at least mention the possible surprising cases where the
> > heuristics fails in this comment.
> 
> Yeah, it's not perfect, but I also failed to come up with a better
> solution... do you have any other ideas?

Maybe fixing the qcow vs. qcow2 case could be possible relatively easily
by operating only on full words, but I don't have any fundamentally
different solution without touching every script.

If touching everything is an option, maybe we could have a standardised
comment line like we have with "# group:" today. This would be easy to
parse and work in both bash and Python scripts. But I'm not sure if it's
worth it.

> If not, would you be fine with the patch if I respin it with the comment
> here extended with some words about those surprising cases?

Yes, I just want to have potentially surprising behaviour documented.

Kevin