Re: [RFC PATCH-for-5.2? 4/5] tests/acceptance: Only run tests tagged 'gating-ci' on GitLab CI

["Philippe Mathieu-Daudé" <philmd@redhat.com>]

On 11/2/20 3:42 PM, Philippe Mathieu-Daudé wrote:
> To avoid breaking our CI each time a test is added, switch from the
> "run all but disable some" to "only run the tagged tests on CI".
> This way we can add a test to the repository, and promote it to
> 'gating-ci' once it is proven stable enough.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
> 
> TODO: where to add documentation?
> 
> - docs/devel/testing.rst (too big, split?)
> - tests/acceptance/README.rst
> 
> ---
>  tests/Makefile.include | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 3a0524ce740..f39ba760c17 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -126,7 +126,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>              $(TESTS_VENV_DIR)/bin/python -m avocado \
>              --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>              --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
> -            $(AVOCADO_TAGS) \
> +            $(AVOCADO_TAGS) -t gating-ci \

This doesn't work as expected, since we have:

AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter
%-softmmu,$(TARGET_DIRS)))

And avocado '-t' works as logical OR, not logical AND.

OTOH it seems this variable predate the auto-skip feature
(when a binary is not present).

So I'll test this instead, which is simpler:

-- >8 --
@@ -90,7 +90,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
 AVOCADO_SHOW=app
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter
%-softmmu,$(TARGET_DIRS)))
+AVOCADO_TAGS=-t gating-ci

 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
        $(call quiet-command, \
---