Re: [RFC PATCH 01/14] meson: Fix meson suites invocation from make check

[Fabiano Rosas <farosas@suse.de>]

Fabiano Rosas <farosas@suse.de> writes:

> make check is currently by default running tests tagged
> "slow". Running make check with no SPEED variable set should be
> equivalent to SPEED=quick and run the meson 'quick' test setup, which
> explicitly excludes slow tests:
>
>   add_test_setup('quick', exclude_suites: ['slow', 'thorough'] ...
>
> The problem is that the mtest2make.py script is effectively overriding
> the "meson test --setup" setting by using --suite which takes
> precedence over --setup according to meson documentation.
>
> Fix by excluding suites directly with the "--no-suite" option while
> generating Makefile.mtest.
>
> Although the 'exclude_suites' in meson.build are useless for make
> check, they still help when running 'meson test' from the
> command-line, so keep them around.
>
> For the configure line I'm using:
>
> make check goes from 1100 to 996 tests
> make SPEED=slow check stays at 2158 tests
> make SPEED=thorough check stays at 2279 tests
>
> (the rest of the test targets look sane, but maintainers please
> double-check)
>
> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> ---
> - With this change we obviously lose coverage in make check. I suggest
>   we use this opportunity to review what gets put in the quick
>   suite.
>
>   For migration, I want to keep only the smoke tests in make
>   check. I'll send a patch adding them back if this change is
>   accepted.
>
> - Some of the suffixes from functional tests could probably be removed
>   and rely on having the tests in the proper suite instead. I.e. use
>   'slow' instead of 'foobar_slow'.
> ---
>  scripts/mtest2make.py          | 11 +++++++++--
>  tests/qemu-iotests/meson.build |  2 +-
>  2 files changed, 10 insertions(+), 3 deletions(-)
>
> diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
> index 383ea68b16..a61a83f775 100644
> --- a/scripts/mtest2make.py
> +++ b/scripts/mtest2make.py
> @@ -26,6 +26,11 @@ def names(self, base):
>  .speed.slow = $(sort $(filter-out %-thorough, $1))
>  .speed.thorough = $(sort $1)
>  
> +.speed_exclude_suites.quick = --no-suite slow --no-suite thorough \
> +                              --no-suite optional
> +.speed_exclude_suites.slow = --no-suite thorough
> +.speed_exclude_suites.thorough =
> +
>  TIMEOUT_MULTIPLIER ?= 1
>  .mtestargs = --no-rebuild -t $(TIMEOUT_MULTIPLIER)
>  ifneq ($(SPEED), quick)
> @@ -34,7 +39,8 @@ def names(self, base):
>  .mtestargs += $(subst -j,--num-processes , $(filter-out -j, $(lastword -j1 $(filter -j%, $(MAKEFLAGS)))))
>  
>  .check.mtestargs = $(MTESTARGS) $(.mtestargs) $(if $(V),--verbose,--print-errorlogs) \
> -    $(foreach s, $(sort $(.check.mtest-suites)), --suite $s)
> +    $(foreach s, $(sort $(.check.mtest-suites)), --suite $s) \
> +    $(.speed_exclude_suites.$(SPEED))
>  .bench.mtestargs = $(MTESTARGS) $(.mtestargs) --benchmark --verbose \
>      $(foreach s, $(sort $(.bench.mtest-suites)), --suite $s)''')
>  
> @@ -102,7 +108,8 @@ def emit_suite(name, suite, prefix):
>          targets += f' {prefix} {prefix}-report.junit.xml'
>      print(f'ifneq ($(filter {targets}, $(MAKECMDGOALS)),)')
>      # for the "base" suite possibly add FOO-slow and FOO-thorough
> -    print(f".{prefix}.mtest-suites += {name} $(call .speed.$(SPEED), {names})")
> +    print(f".{prefix}.mtest-suites += {name} $(call .speed.$(SPEED), {names}) "
> +          ".speed_exclude_suites.$(SPEED)")
>      print(f'endif')
>  
>  targets = {t['id']: [os.path.relpath(f) for f in t['filename']]
> diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
> index 8857f25f29..34cfcdc53b 100644
> --- a/tests/qemu-iotests/meson.build
> +++ b/tests/qemu-iotests/meson.build
> @@ -68,7 +68,7 @@ foreach driver, speed: qemu_iotests_drivers
>    endif
>  
>    # Every driver gets put in the driver specific suite
> -  suites = ['block-' + driver + '-optional']
> +  suites = ['block-' + driver + '-optional', 'optional']
>    # Any driver tagged quick or slow also gets added to slow
>    # otherwise its tagged thorough
>    if speed != 'thorough'

Oops, this one sneaked in, please ignore it. I sent it as a separate
patch already.

https://lore.kernel.org/r/20260918203153.3874488-1-farosas@suse.de