[PATCH 0/2] gitlab-ci: Force 'make check' single-threaded for cross-i686-tci

[PATCH 0/2] gitlab-ci: Force 'make check' single-threaded for cross-i686-tci

[Peter Maydell]

The CI job cross-i686-tci has been persistently flaky for what seems
like years now. Stefan Weil had a look and his conclusion was that
this happens because we run too many tests in parallel, and so
sometimes they starve each other of CPU and time out:
https://lore.kernel.org/qemu-devel/9692cfcb-ef59-4cec-8452-8bfb859e8a6c@weilnetz.de/

To test this theory, force the cross-i686-tci job to run only
one test at a time. 'meson test' has a handy environment
variable to let us set the parallelism. Unfortunately meson
prioritises the command line '--num-processes' argument over
the environment variable if they are both set, so to make
this work we need to adjust our mtest2make script to not
set --num-processes if the MESON_TESTTHREADS envariable is set.

I don't know if this will fix the flakiness, but it seems worth
a try. If it works, we might want to consider whether we should
do something similar across the board -- at the moment we
always use a parallelism of num-cpus + 1, because we use the
same amount of parallelism that we do for the compile step.

thanks
-- PMM

Peter Maydell (2):
  scripts/mtest2make: Honour MESON_TESTTHREADS if it is set
  .gitlab-ci.d/crossbuilds.yml: Force 'make check' single threaded for
    cross-i686-tci

 .gitlab-ci.d/crossbuilds.yml | 3 +++
 scripts/mtest2make.py        | 4 ++++
 2 files changed, 7 insertions(+)

-- 
2.34.1

[PATCH 1/2] scripts/mtest2make: Honour MESON_TESTTHREADS if it is set

[Peter Maydell]

'meson test' has two ways to set how many tests to run in parallel:
 * the command line --num-processes option
 * the environment variable MESON_TESTTHREADS
If both are given, then it prefers the command line option over
the environment variable.

This is an unfortunate order of precedence, because it's frequently
handy to be able to override the default by setting an environment
variable at the top level of a build or CI invocation.  Fortunately,
we can flip the order for our purposes, because we always invoke
'meson test' indirectly.

We currently invoke 'meson test' passing it a --num-processes
argument based on whatever the '-j' option to make was.  Adjust the
logic in mtest2make so that if MESON_TESTTHREADS is set then we
honour that, and don't override it.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
 scripts/mtest2make.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/mtest2make.py b/scripts/mtest2make.py
index eb01a05ddbd..6de0044134c 100644
--- a/scripts/mtest2make.py
+++ b/scripts/mtest2make.py
@@ -32,7 +32,11 @@ def names(self, base):
 ifneq ($(SPEED), quick)
 .mtestargs += --setup $(SPEED)
 endif
+# If the user set MESON_TESTTHREADS then don't override that preference by
+# passing --num-processes to 'meson test'
+ifeq ($(MESON_TESTTHREADS),)
 .mtestargs += $(subst -j,--num-processes , $(filter-out -j, $(lastword -j1 $(filter -j%, $(MAKEFLAGS)))))
+endif
 
 .check.mtestargs = $(MTESTARGS) $(.mtestargs) $(if $(V),--verbose,--print-errorlogs)
 .bench.mtestargs = $(MTESTARGS) $(.mtestargs) --benchmark --verbose''')
-- 
2.34.1

[PATCH 2/2] .gitlab-ci.d/crossbuilds.yml: Force 'make check' single threaded for cross-i686-tci

[Peter Maydell]

The cross-i686-tci CI job is persistently flaky with various tests
hitting timeouts.  One theory for why this is happening is that we're
running too many tests in parallel and so sometimes a test gets
starved of CPU and isn't able to complete within the timeout.

Set the MESON_TESTTHREADS environment variable to 1 for this job;
this will cause 'meson test' to run only one test at a time.

(Note that this relies on the change to meson2make that makes it
honour MESON_TESTTHREADS; otherwise it will have no effect.)

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
Seems worth a try -- if this doesn't have an effect then
we can revert it, but we'll at least have determined what
the problem isn't...
---
 .gitlab-ci.d/crossbuilds.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index cb499e4ee0d..ca1db011b11 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -70,6 +70,9 @@ cross-i686-tci:
     ACCEL: tcg-interpreter
     EXTRA_CONFIGURE_OPTS: --target-list=i386-softmmu,i386-linux-user,aarch64-softmmu,aarch64-linux-user,ppc-softmmu,ppc-linux-user --disable-plugins --disable-kvm
     MAKE_CHECK_ARGS: check check-tcg
+    # Force 'meson test' to run only one test at once, to
+    # see whether this reduces the flakiness of this CI job.
+    MESON_TESTTHREADS: 1
 
 cross-mipsel-system:
   extends: .cross_system_build_job
-- 
2.34.1

Re: [PATCH 2/2] .gitlab-ci.d/crossbuilds.yml: Force 'make check' single threaded for cross-i686-tci

[Thomas Huth]

On 06/09/2024 20.07, Peter Maydell wrote:
> The cross-i686-tci CI job is persistently flaky with various tests
> hitting timeouts.  One theory for why this is happening is that we're
> running too many tests in parallel and so sometimes a test gets
> starved of CPU and isn't able to complete within the timeout.
> 
> Set the MESON_TESTTHREADS environment variable to 1 for this job;
> this will cause 'meson test' to run only one test at a time.
> 
> (Note that this relies on the change to meson2make that makes it
> honour MESON_TESTTHREADS; otherwise it will have no effect.)
> 
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> ---
> Seems worth a try -- if this doesn't have an effect then
> we can revert it, but we'll at least have determined what
> the problem isn't...
> ---
>   .gitlab-ci.d/crossbuilds.yml | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> index cb499e4ee0d..ca1db011b11 100644
> --- a/.gitlab-ci.d/crossbuilds.yml
> +++ b/.gitlab-ci.d/crossbuilds.yml
> @@ -70,6 +70,9 @@ cross-i686-tci:
>       ACCEL: tcg-interpreter
>       EXTRA_CONFIGURE_OPTS: --target-list=i386-softmmu,i386-linux-user,aarch64-softmmu,aarch64-linux-user,ppc-softmmu,ppc-linux-user --disable-plugins --disable-kvm
>       MAKE_CHECK_ARGS: check check-tcg
> +    # Force 'meson test' to run only one test at once, to
> +    # see whether this reduces the flakiness of this CI job.
> +    MESON_TESTTHREADS: 1

Can't we simply add "-j1" to the MAKE_CHECK_ARGS line?

According to the man-page of "make":

  "If there is more than one -j option, the last one is effective."

So adding a -j1 should override the previous setting, I think.

  Thomas

Re: [PATCH 2/2] .gitlab-ci.d/crossbuilds.yml: Force 'make check' single threaded for cross-i686-tci

[Peter Maydell]

On Tue, 10 Sept 2024 at 13:13, Thomas Huth <thuth@redhat.com> wrote:
>
> On 06/09/2024 20.07, Peter Maydell wrote:
> > The cross-i686-tci CI job is persistently flaky with various tests
> > hitting timeouts.  One theory for why this is happening is that we're
> > running too many tests in parallel and so sometimes a test gets
> > starved of CPU and isn't able to complete within the timeout.
> >
> > Set the MESON_TESTTHREADS environment variable to 1 for this job;
> > this will cause 'meson test' to run only one test at a time.
> >
> > (Note that this relies on the change to meson2make that makes it
> > honour MESON_TESTTHREADS; otherwise it will have no effect.)
> >
> > Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> > ---
> > Seems worth a try -- if this doesn't have an effect then
> > we can revert it, but we'll at least have determined what
> > the problem isn't...
> > ---
> >   .gitlab-ci.d/crossbuilds.yml | 3 +++
> >   1 file changed, 3 insertions(+)
> >
> > diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> > index cb499e4ee0d..ca1db011b11 100644
> > --- a/.gitlab-ci.d/crossbuilds.yml
> > +++ b/.gitlab-ci.d/crossbuilds.yml
> > @@ -70,6 +70,9 @@ cross-i686-tci:
> >       ACCEL: tcg-interpreter
> >       EXTRA_CONFIGURE_OPTS: --target-list=i386-softmmu,i386-linux-user,aarch64-softmmu,aarch64-linux-user,ppc-softmmu,ppc-linux-user --disable-plugins --disable-kvm
> >       MAKE_CHECK_ARGS: check check-tcg
> > +    # Force 'meson test' to run only one test at once, to
> > +    # see whether this reduces the flakiness of this CI job.
> > +    MESON_TESTTHREADS: 1
>
> Can't we simply add "-j1" to the MAKE_CHECK_ARGS line?
>
> According to the man-page of "make":
>
>   "If there is more than one -j option, the last one is effective."
>
> So adding a -j1 should override the previous setting, I think.

You would also be relying on the rune in mtest2make to
pick the last -j option in the list, but luckily I think
it already does that:

.mtestargs += $(subst -j,--num-processes , $(filter-out -j, $(lastword
-j1 $(filter -j%, $(MAKEFLAGS)))))

thanks
-- PMM