[PATCH 0/3] gitlab: fix failing TSAN job in CI

[PATCH 0/3] gitlab: fix failing TSAN job in CI

[Daniel P. Berrangé]

See patch 3 commit message for the key details about the problem.

Daniel P. Berrangé (3):
  dockerfiles: add 'MAKE' env variable to remaining containers
  gitlab: use $MAKE instead of 'make'
  gitlab: use 'setarch -R' to workaround tsan bug

 .gitlab-ci.d/buildtest-template.yml                      | 6 +++---
 .gitlab-ci.d/buildtest.yml                               | 3 +++
 tests/docker/dockerfiles/debian-all-test-cross.docker    | 1 +
 tests/docker/dockerfiles/debian-hexagon-cross.docker     | 1 +
 tests/docker/dockerfiles/debian-legacy-test-cross.docker | 1 +
 tests/docker/dockerfiles/debian-loongarch-cross.docker   | 1 +
 tests/docker/dockerfiles/debian-tricore-cross.docker     | 1 +
 tests/docker/dockerfiles/debian-xtensa-cross.docker      | 1 +
 tests/docker/dockerfiles/fedora-cris-cross.docker        | 1 +
 9 files changed, 13 insertions(+), 3 deletions(-)

-- 
2.43.0

[PATCH 1/3] dockerfiles: add 'MAKE' env variable to remaining containers

[Daniel P. Berrangé]

All the lcitool generated containers define a "MAKE" env. It will be
convenient for later patches if all containers do this.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/docker/dockerfiles/debian-all-test-cross.docker    | 1 +
 tests/docker/dockerfiles/debian-hexagon-cross.docker     | 1 +
 tests/docker/dockerfiles/debian-legacy-test-cross.docker | 1 +
 tests/docker/dockerfiles/debian-loongarch-cross.docker   | 1 +
 tests/docker/dockerfiles/debian-tricore-cross.docker     | 1 +
 tests/docker/dockerfiles/debian-xtensa-cross.docker      | 1 +
 tests/docker/dockerfiles/fedora-cris-cross.docker        | 1 +
 7 files changed, 7 insertions(+)

diff --git a/tests/docker/dockerfiles/debian-all-test-cross.docker b/tests/docker/dockerfiles/debian-all-test-cross.docker
index 2cc7a24d4d..6cc38a3633 100644
--- a/tests/docker/dockerfiles/debian-all-test-cross.docker
+++ b/tests/docker/dockerfiles/debian-all-test-cross.docker
@@ -68,6 +68,7 @@ RUN DEBIAN_FRONTEND=noninteractive eatmydata \
 ENV QEMU_CONFIGURE_OPTS --disable-system --disable-docs --disable-tools
 ENV DEF_TARGET_LIST aarch64-linux-user,arm-linux-user,hppa-linux-user,i386-linux-user,m68k-linux-user,mips-linux-user,mips64-linux-user,mips64el-linux-user,mipsel-linux-user,ppc-linux-user,ppc64-linux-user,ppc64le-linux-user,riscv64-linux-user,s390x-linux-user,sparc64-linux-user
 # As a final step configure the user (if env is defined)
+ENV MAKE /usr/bin/make
 ARG USER
 ARG UID
 RUN if [ "${USER}" ]; then \
diff --git a/tests/docker/dockerfiles/debian-hexagon-cross.docker b/tests/docker/dockerfiles/debian-hexagon-cross.docker
index 60bd8faa20..f2d40f2dee 100644
--- a/tests/docker/dockerfiles/debian-hexagon-cross.docker
+++ b/tests/docker/dockerfiles/debian-hexagon-cross.docker
@@ -45,6 +45,7 @@ ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
 
 RUN curl -#SL "$TOOLCHAIN_URL" | tar -xJC "$TOOLCHAIN_INSTALL"
 ENV PATH $PATH:${TOOLCHAIN_INSTALL}/${TOOLCHAIN_BASENAME}/x86_64-linux-gnu/bin
+ENV MAKE /usr/bin/make
 # As a final step configure the user (if env is defined)
 ARG USER
 ARG UID
diff --git a/tests/docker/dockerfiles/debian-legacy-test-cross.docker b/tests/docker/dockerfiles/debian-legacy-test-cross.docker
index 8cc68bc912..d75e0b85e2 100644
--- a/tests/docker/dockerfiles/debian-legacy-test-cross.docker
+++ b/tests/docker/dockerfiles/debian-legacy-test-cross.docker
@@ -42,6 +42,7 @@ RUN /usr/bin/pip3 install tomli
 
 ENV QEMU_CONFIGURE_OPTS --disable-system --disable-docs --disable-tools
 ENV DEF_TARGET_LIST alpha-linux-user,sh4-linux-user
+ENV MAKE /usr/bin/make
 # As a final step configure the user (if env is defined)
 ARG USER
 ARG UID
diff --git a/tests/docker/dockerfiles/debian-loongarch-cross.docker b/tests/docker/dockerfiles/debian-loongarch-cross.docker
index b25e779a2c..6a9197528b 100644
--- a/tests/docker/dockerfiles/debian-loongarch-cross.docker
+++ b/tests/docker/dockerfiles/debian-loongarch-cross.docker
@@ -44,6 +44,7 @@ ENV LD_LIBRARY_PATH /opt/cross-tools/lib:/opt/cross-tools/loongarch64-unknown-li
 
 ENV QEMU_CONFIGURE_OPTS --disable-system --disable-docs --disable-tools
 ENV DEF_TARGET_LIST loongarch64-linux-user,loongarch-softmmu
+ENV MAKE /usr/bin/make
 
 # As a final step configure the user (if env is defined)
 ARG USER
diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker
index c597f8e16b..16276aa21d 100644
--- a/tests/docker/dockerfiles/debian-tricore-cross.docker
+++ b/tests/docker/dockerfiles/debian-tricore-cross.docker
@@ -44,6 +44,7 @@ RUN curl -#SL https://github.com/bkoppelmann/package_940/releases/download/trico
 # This image can only build a very minimal QEMU as well as the tests
 ENV DEF_TARGET_LIST tricore-softmmu
 ENV QEMU_CONFIGURE_OPTS --disable-user --disable-tools --disable-fdt
+ENV MAKE /usr/bin/make
 # As a final step configure the user (if env is defined)
 ARG USER
 ARG UID
diff --git a/tests/docker/dockerfiles/debian-xtensa-cross.docker b/tests/docker/dockerfiles/debian-xtensa-cross.docker
index 72c25d63d9..413881899b 100644
--- a/tests/docker/dockerfiles/debian-xtensa-cross.docker
+++ b/tests/docker/dockerfiles/debian-xtensa-cross.docker
@@ -27,6 +27,7 @@ RUN for cpu in $CPU_LIST; do \
     done
 
 ENV PATH $PATH:/opt/$TOOLCHAIN_RELEASE/xtensa-dc232b-elf/bin:/opt/$TOOLCHAIN_RELEASE/xtensa-dc233c-elf/bin:/opt/$TOOLCHAIN_RELEASE/xtensa-de233_fpu-elf/bin:/opt/$TOOLCHAIN_RELEASE/xtensa-dsp3400-elf/bin
+ENV MAKE /usr/bin/make
 # As a final step configure the user (if env is defined)
 ARG USER
 ARG UID
diff --git a/tests/docker/dockerfiles/fedora-cris-cross.docker b/tests/docker/dockerfiles/fedora-cris-cross.docker
index f2899af410..97c9d37ede 100644
--- a/tests/docker/dockerfiles/fedora-cris-cross.docker
+++ b/tests/docker/dockerfiles/fedora-cris-cross.docker
@@ -4,6 +4,7 @@
 
 FROM registry.fedoraproject.org/fedora:33
 ENV PACKAGES gcc-cris-linux-gnu
+ENV MAKE /usr/bin/make
 RUN dnf install -y $PACKAGES
 RUN rpm -q $PACKAGES | sort > /packages.txt
 # As a final step configure the user (if env is defined)
-- 
2.43.0

Re: [PATCH 1/3] dockerfiles: add 'MAKE' env variable to remaining containers

[Thomas Huth]

On 13/05/2024 13.15, Daniel P. Berrangé wrote:
> All the lcitool generated containers define a "MAKE" env. It will be
> convenient for later patches if all containers do this.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   tests/docker/dockerfiles/debian-all-test-cross.docker    | 1 +
>   tests/docker/dockerfiles/debian-hexagon-cross.docker     | 1 +
>   tests/docker/dockerfiles/debian-legacy-test-cross.docker | 1 +
>   tests/docker/dockerfiles/debian-loongarch-cross.docker   | 1 +
>   tests/docker/dockerfiles/debian-tricore-cross.docker     | 1 +
>   tests/docker/dockerfiles/debian-xtensa-cross.docker      | 1 +
>   tests/docker/dockerfiles/fedora-cris-cross.docker        | 1 +
>   7 files changed, 7 insertions(+)

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 1/3] dockerfiles: add 'MAKE' env variable to remaining containers

[Philippe Mathieu-Daudé]

On 13/5/24 13:15, Daniel P. Berrangé wrote:
> All the lcitool generated containers define a "MAKE" env. It will be
> convenient for later patches if all containers do this.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   tests/docker/dockerfiles/debian-all-test-cross.docker    | 1 +
>   tests/docker/dockerfiles/debian-hexagon-cross.docker     | 1 +
>   tests/docker/dockerfiles/debian-legacy-test-cross.docker | 1 +
>   tests/docker/dockerfiles/debian-loongarch-cross.docker   | 1 +
>   tests/docker/dockerfiles/debian-tricore-cross.docker     | 1 +
>   tests/docker/dockerfiles/debian-xtensa-cross.docker      | 1 +
>   tests/docker/dockerfiles/fedora-cris-cross.docker        | 1 +
>   7 files changed, 7 insertions(+)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

[PATCH 2/3] gitlab: use $MAKE instead of 'make'

[Daniel P. Berrangé]

The lcitool generated containers have '$MAKE' set to the path
of the right 'make' binary. Using the env variable makes it
possible to override the choice per job.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/buildtest-template.yml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index 22045add80..278a5ea966 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -26,10 +26,10 @@
       then
         pyvenv/bin/meson configure . -Dbackend_max_links="$LD_JOBS" ;
       fi || exit 1;
-    - make -j"$JOBS"
+    - $MAKE -j"$JOBS"
     - if test -n "$MAKE_CHECK_ARGS";
       then
-        make -j"$JOBS" $MAKE_CHECK_ARGS ;
+        $MAKE -j"$JOBS" $MAKE_CHECK_ARGS ;
       fi
     - ccache --show-stats
 
@@ -60,7 +60,7 @@
     - cd build
     - find . -type f -exec touch {} +
     # Avoid recompiling by hiding ninja with NINJA=":"
-    - make NINJA=":" $MAKE_CHECK_ARGS
+    - $MAKE NINJA=":" $MAKE_CHECK_ARGS
 
 .native_test_job_template:
   extends: .common_test_job_template
-- 
2.43.0

Re: [PATCH 2/3] gitlab: use $MAKE instead of 'make'

[Thomas Huth]

On 13/05/2024 13.15, Daniel P. Berrangé wrote:
> The lcitool generated containers have '$MAKE' set to the path
> of the right 'make' binary. Using the env variable makes it
> possible to override the choice per job.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 2/3] gitlab: use $MAKE instead of 'make'

[Philippe Mathieu-Daudé]

On 13/5/24 13:15, Daniel P. Berrangé wrote:
> The lcitool generated containers have '$MAKE' set to the path
> of the right 'make' binary. Using the env variable makes it
> possible to override the choice per job.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

[PATCH 3/3] gitlab: use 'setarch -R' to workaround tsan bug

[Daniel P. Berrangé]

The TSAN job started failing when gitlab rolled out their latest
release. The root cause is a change in the Google COS version used
on shared runners. This brings a kernel running with

 vm.mmap_rnd_bits = 31

which is incompatible with TSAN in LLVM < 18, which only supports
upto '28'. LLVM 18 can support upto '30', and failing that will
re-exec itself to turn off VA randomization.

Our LLVM is too old for now, but we can run with 'setarch -R make ..'
to turn off VA randomization ourselves.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/buildtest.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index bab6194564..d864562628 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -575,6 +575,9 @@ tsan-build:
     CONFIGURE_ARGS: --enable-tsan --cc=clang --cxx=clang++
           --enable-trace-backends=ust --disable-slirp
     TARGETS: x86_64-softmmu ppc64-softmmu riscv64-softmmu x86_64-linux-user
+    # Remove when we switch to a distro with clang >= 18
+    # https://github.com/google/sanitizers/issues/1716
+    MAKE: setarch -R make
 
 # gcov is a GCC features
 gcov:
-- 
2.43.0

Re: [PATCH 3/3] gitlab: use 'setarch -R' to workaround tsan bug

[Thomas Huth]

On 13/05/2024 13.15, Daniel P. Berrangé wrote:
> The TSAN job started failing when gitlab rolled out their latest
> release. The root cause is a change in the Google COS version used
> on shared runners. This brings a kernel running with
> 
>   vm.mmap_rnd_bits = 31
> 
> which is incompatible with TSAN in LLVM < 18, which only supports
> upto '28'. LLVM 18 can support upto '30', and failing that will
> re-exec itself to turn off VA randomization.
> 
> Our LLVM is too old for now, but we can run with 'setarch -R make ..'
> to turn off VA randomization ourselves.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/buildtest.yml | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index bab6194564..d864562628 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -575,6 +575,9 @@ tsan-build:
>       CONFIGURE_ARGS: --enable-tsan --cc=clang --cxx=clang++
>             --enable-trace-backends=ust --disable-slirp
>       TARGETS: x86_64-softmmu ppc64-softmmu riscv64-softmmu x86_64-linux-user
> +    # Remove when we switch to a distro with clang >= 18
> +    # https://github.com/google/sanitizers/issues/1716
> +    MAKE: setarch -R make

Thanks for tackling this!

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 3/3] gitlab: use 'setarch -R' to workaround tsan bug

[Daniel P. Berrangé]

On Mon, May 13, 2024 at 12:15:51PM +0100, Daniel P. Berrangé wrote:
> The TSAN job started failing when gitlab rolled out their latest
> release. The root cause is a change in the Google COS version used
> on shared runners. This brings a kernel running with
> 
>  vm.mmap_rnd_bits = 31
> 
> which is incompatible with TSAN in LLVM < 18, which only supports
> upto '28'. LLVM 18 can support upto '30', and failing that will
> re-exec itself to turn off VA randomization.
> 
> Our LLVM is too old for now, but we can run with 'setarch -R make ..'
> to turn off VA randomization ourselves.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/buildtest.yml | 3 +++
>  1 file changed, 3 insertions(+)

Example job showing this working:

  https://gitlab.com/berrange/qemu/-/jobs/6824465594

 > 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index bab6194564..d864562628 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -575,6 +575,9 @@ tsan-build:
>      CONFIGURE_ARGS: --enable-tsan --cc=clang --cxx=clang++
>            --enable-trace-backends=ust --disable-slirp
>      TARGETS: x86_64-softmmu ppc64-softmmu riscv64-softmmu x86_64-linux-user
> +    # Remove when we switch to a distro with clang >= 18
> +    # https://github.com/google/sanitizers/issues/1716
> +    MAKE: setarch -R make
>  
>  # gcov is a GCC features
>  gcov:
> -- 
> 2.43.0
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|