[RFC PATCH] tests/tcg: skip the vma-pthread test on CI

[RFC PATCH] tests/tcg: skip the vma-pthread test on CI

[Alex Bennée]

We are getting a lot of failures that are not related to changes so
this could be a flaky test.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Richard Henderson <richard.henderson@linaro.org>
---
 tests/tcg/multiarch/Makefile.target | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tests/tcg/multiarch/Makefile.target b/tests/tcg/multiarch/Makefile.target
index e7213af492..ae8b3d7268 100644
--- a/tests/tcg/multiarch/Makefile.target
+++ b/tests/tcg/multiarch/Makefile.target
@@ -42,6 +42,15 @@ munmap-pthread: LDFLAGS+=-pthread
 vma-pthread: CFLAGS+=-pthread
 vma-pthread: LDFLAGS+=-pthread
 
+# The vma-pthread seems very sensitive on gitlab and we currently
+# don't know if its exposing a real bug or the test is flaky.
+ifneq ($(GITLAB_CI),)
+run-vma-pthread: vma-pthread
+	$(call skip-test, $<, "flaky on CI?")
+run-plugin-vma-pthread-with-%: vma-pthread
+	$(call skip-test, $<, "flaky on CI?")
+endif
+
 # We define the runner for test-mmap after the individual
 # architectures have defined their supported pages sizes. If no
 # additional page sizes are defined we only run the default test.
-- 
2.34.1

Re: [RFC PATCH] tests/tcg: skip the vma-pthread test on CI

[Richard Henderson]

On 1/16/23 07:32, Alex Bennée wrote:
> diff --git a/tests/tcg/multiarch/Makefile.target b/tests/tcg/multiarch/Makefile.target
> index e7213af492..ae8b3d7268 100644
> --- a/tests/tcg/multiarch/Makefile.target
> +++ b/tests/tcg/multiarch/Makefile.target
> @@ -42,6 +42,15 @@ munmap-pthread: LDFLAGS+=-pthread
>   vma-pthread: CFLAGS+=-pthread
>   vma-pthread: LDFLAGS+=-pthread
>   
> +# The vma-pthread seems very sensitive on gitlab and we currently
> +# don't know if its exposing a real bug or the test is flaky.
> +ifneq ($(GITLAB_CI),)
> +run-vma-pthread: vma-pthread
> +	$(call skip-test, $<, "flaky on CI?")
> +run-plugin-vma-pthread-with-%: vma-pthread
> +	$(call skip-test, $<, "flaky on CI?")
> +endif
> +

Ok I guess.  I'd have thought the ifdef around the entire mention of the test would be 
better -- no point in even building it.  But,

Acked-by: Richard Henderson <richard.henderson@linaro.org>


r~

Re: [RFC PATCH] tests/tcg: skip the vma-pthread test on CI

[Alex Bennée]

[-- Attachment #1: Type: text/plain, Size: 1386 bytes --]

I did consider it but it would involve messing about with filter to remove
the test from the wildcards. This way we don't forget about it when looking
through the logs.

I've not been able to get this to fail on any other machine though. It's
been rock solid over several thousand runs.

On Mon, 16 Jan 2023, 19:25 Richard Henderson, <richard.henderson@linaro.org>
wrote:

> On 1/16/23 07:32, Alex Bennée wrote:
> > diff --git a/tests/tcg/multiarch/Makefile.target
> b/tests/tcg/multiarch/Makefile.target
> > index e7213af492..ae8b3d7268 100644
> > --- a/tests/tcg/multiarch/Makefile.target
> > +++ b/tests/tcg/multiarch/Makefile.target
> > @@ -42,6 +42,15 @@ munmap-pthread: LDFLAGS+=-pthread
> >   vma-pthread: CFLAGS+=-pthread
> >   vma-pthread: LDFLAGS+=-pthread
> >
> > +# The vma-pthread seems very sensitive on gitlab and we currently
> > +# don't know if its exposing a real bug or the test is flaky.
> > +ifneq ($(GITLAB_CI),)
> > +run-vma-pthread: vma-pthread
> > +     $(call skip-test, $<, "flaky on CI?")
> > +run-plugin-vma-pthread-with-%: vma-pthread
> > +     $(call skip-test, $<, "flaky on CI?")
> > +endif
> > +
>
> Ok I guess.  I'd have thought the ifdef around the entire mention of the
> test would be
> better -- no point in even building it.  But,
>
> Acked-by: Richard Henderson <richard.henderson@linaro.org>
>
>
> r~
>

[-- Attachment #2: Type: text/html, Size: 1943 bytes --]