[PATCH v5 00/17] testing: various docker clean-ups post meson tcg conversion (pre-PR)

[PATCH v5 00/17] testing: various docker clean-ups post meson tcg conversion (pre-PR)

[Alex Bennée]

This cleans up a few bits and pieces although doesn't totally solve
the issue of an apparent stall while meson builds the docker image.
However with these changes there should be no difference between the
meson build and the equivalent "make docker-image-foo" invocation.

We also finally get rid of the ill-advised magic caching support in
docker.py.

v5
 - fixup a bunch of rebase damage
 - swap out my ccache fix for Anders
 - rename revert to "configure: ensure it also honours container configuration"
 - use Pierricks solution for avoid too many SKIPs while validating config

v4
 - added Illya's fixes for out of tree and s390x tests
 - tweaked the revert so --disable-containers still passed to meson
 - don't define tests if no compiler available to avoid excess SKIPs

v3

 - Update make2test to trigger docker build from Makefile.include.
 - add Daniels's suggested console: true patch
 - fixed up some unref vars from the reverts

v2

I've restored the configure based compiler checks for so we can build
roms and the vdso blobs again. It looks like we could never build roms
with container based images so I've left it for now. Unlike the vdso
build we can't trigger a container build through Makefile.prereqs.

TODO
----

I also need to look at the timestamp stuff that Pierrick has suggested
to see if that improves the caching from the registry. However that
can wait until after the PR.

The following still need review:

  gitlab: drop check-tcg from clang-system job
  tests/tcg: don't define tests when missing compilers

Please test and review.

Alex.

Alex Bennée (13):
  Revert "configure: remove cross_cc and gdb detection logic"
  configure: ensure it also honours container configuration
  configure: re-purpose the tcg test compiler configs for vdso building
  tests/tcg: prefix meson built docker tags with qemu/
  tests/tcg: use --add-current-user when building containers
  tests/tcg: don't define tests when missing compilers
  tests/docker: add container registry to configure and use
  tests/docker: remove used _read_qemu_dockerfile
  tests/docker: strip remaining caching logic from docker.py
  tests/docker: add specific command to parser
  gitlab: update the @user id mapping file
  tests/tcg: add console property to docker image custom target
  gitlab: drop check-tcg from clang-system job

Anders Roxell (1):
  tests/docker: fix ccache directory variable expansion

Daniel P. Berrangé (1):
  configure: correctly honour --disable-containers

Ilya Leoshkevich (2):
  tests/tcg/s390x: Provide the libc bits needed by the system tests
  tests/tcg: Fix building outside of the source tree

 configure                               | 465 +++++++++++++++++++++++-
 tests/tcg/s390x/libc/stdio.h            |  11 +
 tests/tcg/s390x/libc/string.h           |  12 +
 .gitlab-ci.d/buildtest.yml              |   2 +-
 .gitlab-map-auto                        |   8 +-
 linux-user/aarch64/Makefile.vdso        |   2 +-
 linux-user/arm/Makefile.vdso            |   2 +-
 linux-user/hppa/Makefile.vdso           |   2 +-
 linux-user/i386/Makefile.vdso           |   2 +-
 linux-user/loongarch64/Makefile.vdso    |   2 +-
 linux-user/ppc/Makefile.vdso            |   2 +-
 linux-user/riscv/Makefile.vdso          |   2 +-
 linux-user/s390x/Makefile.vdso          |   2 +-
 linux-user/sh4/Makefile.vdso            |   2 +-
 linux-user/x86_64/Makefile.vdso         |   2 +-
 meson_options.txt                       |   3 +
 scripts/coverity-scan/run-coverity-scan |   2 +-
 scripts/gitlab-map-update               |   2 +-
 scripts/meson-buildoptions.sh           |   3 +
 tests/docker/Makefile.include           |   6 +-
 tests/docker/docker.py                  | 172 +++------
 tests/tcg/meson.build                   |  30 +-
 tests/tcg/s390x/system/meson.build      |   2 +
 23 files changed, 591 insertions(+), 147 deletions(-)
 create mode 100644 tests/tcg/s390x/libc/stdio.h
 create mode 100644 tests/tcg/s390x/libc/string.h

-- 
2.47.3

[PATCH v5 01/17] Revert "configure: remove cross_cc and gdb detection logic"

[Alex Bennée]

We want to restore the logic to configure as it's also used by the
firmware builds.

This reverts commit 031f935567ab8abfa4e84682ecea08915ca9656d.

Tested-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 configure | 461 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 461 insertions(+)

diff --git a/configure b/configure
index aaa41fe47aa..1973f023b1e 100755
--- a/configure
+++ b/configure
@@ -1266,6 +1266,366 @@ EOF
   fi
 fi
 
+##########################################
+# functions to probe cross compilers
+
+if test "$container_command" = ""; then
+    container_command=$($python "$source_path"/tests/docker/docker.py probe)
+    test "$container_command" = "no" && container_command=""
+fi
+if test $use_containers = "yes" && test "$container_command" != ""; then
+    docker_py="$python $source_path/tests/docker/docker.py --command $container_command"
+fi
+
+# cross compilers defaults, can be overridden with --cross-cc-ARCH
+: ${cross_prefix_aarch64="aarch64-linux-gnu-"}
+: ${cross_prefix_aarch64_be="$cross_prefix_aarch64"}
+: ${cross_prefix_alpha="alpha-linux-gnu-"}
+: ${cross_prefix_arm="arm-linux-gnueabihf-"}
+: ${cross_prefix_armeb="$cross_prefix_arm"}
+: ${cross_prefix_hexagon="hexagon-unknown-linux-musl-"}
+: ${cross_prefix_loongarch64="loongarch64-unknown-linux-gnu-"}
+: ${cross_prefix_hppa="hppa-linux-gnu-"}
+: ${cross_prefix_i386="i686-linux-gnu-"}
+: ${cross_prefix_m68k="m68k-linux-gnu-"}
+: ${cross_prefix_microblaze="microblaze-linux-musl-"}
+: ${cross_prefix_mips64el="mips64el-linux-gnuabi64-"}
+: ${cross_prefix_mips64="mips64-linux-gnuabi64-"}
+: ${cross_prefix_mipsel="mipsel-linux-gnu-"}
+: ${cross_prefix_mips="mips-linux-gnu-"}
+: ${cross_prefix_ppc="powerpc-linux-gnu-"}
+: ${cross_prefix_ppc64="powerpc64-linux-gnu-"}
+: ${cross_prefix_ppc64le="$cross_prefix_ppc64"}
+: ${cross_prefix_riscv64="riscv64-linux-gnu-"}
+: ${cross_prefix_s390x="s390x-linux-gnu-"}
+: ${cross_prefix_sh4="sh4-linux-gnu-"}
+: ${cross_prefix_sparc64="sparc64-linux-gnu-"}
+: ${cross_prefix_sparc="$cross_prefix_sparc64"}
+: ${cross_prefix_tricore="tricore-"}
+: ${cross_prefix_x86_64="x86_64-linux-gnu-"}
+
+: ${cross_cc_aarch64_be="$cross_cc_aarch64"}
+: ${cross_cc_cflags_aarch64_be="-mbig-endian"}
+: ${cross_cc_armeb="$cross_cc_arm"}
+: ${cross_cc_cflags_armeb="-mbig-endian"}
+: ${cross_cc_hexagon="hexagon-unknown-linux-musl-clang"}
+: ${cross_cc_cflags_hexagon="-mv73 -O2 -static"}
+: ${cross_cc_cflags_i386="-m32"}
+: ${cross_cc_cflags_ppc="-m32 -mbig-endian"}
+: ${cross_cc_cflags_ppc64="-m64 -mbig-endian"}
+: ${cross_cc_ppc64le="$cross_cc_ppc64"}
+: ${cross_cc_cflags_ppc64le="-m64 -mlittle-endian"}
+: ${cross_cc_cflags_sparc64="-m64 -mcpu=ultrasparc"}
+: ${cross_cc_sparc="$cross_cc_sparc64"}
+: ${cross_cc_cflags_sparc="-m32 -mcpu=supersparc"}
+: ${cross_cc_cflags_x86_64="-m64 -mcx16"}
+
+compute_target_variable() {
+  eval "$2="
+  if eval test -n "\"\${cross_prefix_$1}\""; then
+    if eval has "\"\${cross_prefix_$1}\$3\""; then
+      eval "$2=\"\${cross_prefix_$1}\$3\""
+    fi
+  fi
+}
+
+have_target() {
+  for i; do
+    case " $target_list " in
+      *" $i "*) return 0;;
+      *) ;;
+    esac
+  done
+  return 1
+}
+
+# probe_target_compiler TARGET
+#
+# Look for a compiler for the given target, either native or cross.
+# Set variables target_* if a compiler is found, and container_cross_*
+# if a Docker-based cross-compiler image is known for the target.
+# Set got_cross_cc to yes/no depending on whether a non-container-based
+# compiler was found.
+#
+# If TARGET is a user-mode emulation target, also set build_static to
+# "y" if static linking is possible.
+#
+probe_target_compiler() {
+  # reset all output variables
+  got_cross_cc=no
+  container_image=
+  container_hosts=
+  container_cross_prefix=
+  container_cross_cc=
+  container_cross_ar=
+  container_cross_as=
+  container_cross_ld=
+  container_cross_nm=
+  container_cross_objcopy=
+  container_cross_ranlib=
+  container_cross_strip=
+
+  target_arch=${1%%-*}
+  case $target_arch in
+    aarch64) container_hosts="x86_64 aarch64" ;;
+    aarch64_be) container_hosts="x86_64 aarch64" ;;
+    alpha) container_hosts=x86_64 ;;
+    arm) container_hosts="x86_64 aarch64" ;;
+    hexagon) container_hosts=x86_64 ;;
+    hppa) container_hosts=x86_64 ;;
+    i386) container_hosts=x86_64 ;;
+    loongarch64) container_hosts=x86_64 ;;
+    m68k) container_hosts=x86_64 ;;
+    microblaze) container_hosts=x86_64 ;;
+    mips64el) container_hosts=x86_64 ;;
+    mips64) container_hosts=x86_64 ;;
+    mipsel) container_hosts=x86_64 ;;
+    mips) container_hosts=x86_64 ;;
+    ppc) container_hosts=x86_64 ;;
+    ppc64|ppc64le) container_hosts=x86_64 ;;
+    riscv64) container_hosts=x86_64 ;;
+    s390x) container_hosts=x86_64 ;;
+    sh4) container_hosts=x86_64 ;;
+    sparc64) container_hosts=x86_64 ;;
+    tricore) container_hosts=x86_64 ;;
+    x86_64) container_hosts="aarch64 ppc64le x86_64" ;;
+    xtensa*) container_hosts=x86_64 ;;
+  esac
+
+  for host in $container_hosts; do
+    test "$container_command" != "" || continue
+    test "$host" = "$cpu" || continue
+    case $target_arch in
+      # debian-all-test-cross architectures
+
+      aarch64_be)
+        container_image=debian-all-test-cross
+        container_cross_prefix=aarch64-linux-gnu-
+        ;;
+      alpha|hppa|m68k|mips|mipsel|riscv64|sh4|sparc64)
+        container_image=debian-all-test-cross
+        ;;
+      mips64)
+        container_image=debian-all-test-cross
+        container_cross_prefix=mips64-linux-gnuabi64-
+        ;;
+      ppc|ppc64|ppc64le)
+        container_image=debian-all-test-cross
+        container_cross_prefix=powerpc${target_arch#ppc}-linux-gnu-
+        ;;
+
+      # architectures with individual containers
+
+      aarch64)
+        # We don't have any bigendian build tools so we only use this for AArch64
+        container_image=debian-arm64-cross
+        ;;
+      arm)
+        # We don't have any bigendian build tools so we only use this for ARM
+        container_image=debian-armhf-cross
+        container_cross_prefix=arm-linux-gnueabihf-
+        ;;
+      hexagon)
+        container_cross_prefix=hexagon-unknown-linux-musl-
+        container_cross_cc=${container_cross_prefix}clang
+        ;;
+      i386)
+        container_image=debian-i686-cross
+        container_cross_prefix=i686-linux-gnu-
+        ;;
+      loongarch64)
+        container_image=debian-loongarch-cross
+        container_cross_prefix=loongarch64-unknown-linux-gnu-
+        ;;
+      microblaze)
+        container_cross_prefix=microblaze-linux-musl-
+        ;;
+      mips64el)
+        container_image=debian-all-test-cross
+        container_cross_prefix=mips64el-linux-gnuabi64-
+        ;;
+      tricore)
+        container_image=debian-tricore-cross
+        container_cross_prefix=tricore-
+        ;;
+      x86_64)
+        container_image=debian-amd64-cross
+        ;;
+      xtensa*)
+        container_image=debian-xtensa-cross
+
+        # default to the dc232b cpu
+        container_cross_prefix=/opt/2020.07/xtensa-dc232b-elf/bin/xtensa-dc232b-elf-
+        ;;
+    esac
+    # Debian and GNU architecture names usually match
+    : ${container_image:=debian-$target_arch-cross}
+    : ${container_cross_prefix:=$target_arch-linux-gnu-}
+    : ${container_cross_cc:=${container_cross_prefix}gcc}
+    : ${container_cross_ar:=${container_cross_prefix}ar}
+    : ${container_cross_as:=${container_cross_prefix}as}
+    : ${container_cross_ld:=${container_cross_prefix}ld}
+    : ${container_cross_nm:=${container_cross_prefix}nm}
+    : ${container_cross_objcopy:=${container_cross_prefix}objcopy}
+    : ${container_cross_ranlib:=${container_cross_prefix}ranlib}
+    : ${container_cross_strip:=${container_cross_prefix}strip}
+  done
+
+  try=cross
+  # For softmmu/roms also look for a bi-endian or multilib-enabled host compiler
+  if [ "${1%softmmu}" != "$1" ] || test "$target_arch" = "$cpu"; then
+      case "$target_arch:$cpu" in
+        aarch64_be:aarch64 | \
+        armeb:arm | \
+        i386:x86_64 | \
+        ppc*:ppc64 | \
+        sparc:sparc64 | \
+        "$cpu:$cpu")
+        try='native cross' ;;
+      esac
+  fi
+  eval "target_cflags=\${cross_cc_cflags_$target_arch}"
+  for thistry in $try; do
+    case $thistry in
+    native)
+      target_cc=$cc
+      target_ccas=$ccas
+      target_ar=$ar
+      target_as=$as
+      target_ld=$ld
+      target_nm=$nm
+      target_objcopy=$objcopy
+      target_ranlib=$ranlib
+      target_strip=$strip
+      ;;
+    cross)
+      target_cc=
+      if eval test -n "\"\${cross_cc_$target_arch}\""; then
+        if eval has "\"\${cross_cc_$target_arch}\""; then
+          eval "target_cc=\"\${cross_cc_$target_arch}\""
+        fi
+      else
+        compute_target_variable $target_arch target_cc gcc
+      fi
+      target_ccas=$target_cc
+      compute_target_variable $target_arch target_ar ar
+      compute_target_variable $target_arch target_as as
+      compute_target_variable $target_arch target_ld ld
+      compute_target_variable $target_arch target_nm nm
+      compute_target_variable $target_arch target_objcopy objcopy
+      compute_target_variable $target_arch target_ranlib ranlib
+      compute_target_variable $target_arch target_strip strip
+      ;;
+    esac
+
+    if test -n "$target_cc"; then
+      case $target_arch in
+        i386|x86_64)
+          if $target_cc --version | grep -qi "clang"; then
+            continue
+          fi
+          ;;
+      esac
+    elif test -n "$target_as" && test -n "$target_ld"; then
+      # Special handling for assembler only targets
+      case $target in
+        tricore-softmmu)
+          build_static=
+          got_cross_cc=yes
+          break
+          ;;
+        *)
+          continue
+          ;;
+      esac
+    else
+      continue
+    fi
+
+    write_c_skeleton
+    case $1 in
+      *-softmmu)
+        if do_compiler "$target_cc" $target_cflags -o $TMPO -c $TMPC &&
+          do_compiler "$target_cc" $target_cflags -r -nostdlib -o "${TMPDIR1}/${TMPB}2.o" "$TMPO" -lgcc; then
+          got_cross_cc=yes
+          break
+        fi
+        ;;
+      *)
+        if do_compiler "$target_cc" $target_cflags -o $TMPE $TMPC -static ; then
+          build_static=y
+          got_cross_cc=yes
+          break
+        fi
+        if do_compiler "$target_cc" $target_cflags -o $TMPE $TMPC ; then
+          build_static=
+          got_cross_cc=yes
+          break
+        fi
+        ;;
+    esac
+  done
+  if test $got_cross_cc != yes; then
+    build_static=
+    target_cc=
+    target_ccas=
+    target_ar=
+    target_as=
+    target_ld=
+    target_nm=
+    target_objcopy=
+    target_ranlib=
+    target_strip=
+  fi
+  test -n "$target_cc"
+}
+
+write_target_makefile() {
+  echo "PYTHON=$python"
+  echo "EXTRA_CFLAGS=$target_cflags"
+  if test -z "$target_cc" && test -z "$target_as"; then
+    test -z "$container_image" && error_exit "Internal error: could not find cross compiler for $1?"
+    echo "$1: docker-image-$container_image" >> Makefile.prereqs
+    if test -n "$container_cross_cc"; then
+      echo "CC=$docker_py cc --cc $container_cross_cc -i qemu/$container_image -s $source_path --"
+      echo "CCAS=$docker_py cc --cc $container_cross_cc -i qemu/$container_image -s $source_path --"
+    fi
+    echo "AR=$docker_py cc --cc $container_cross_ar -i qemu/$container_image -s $source_path --"
+    echo "AS=$docker_py cc --cc $container_cross_as -i qemu/$container_image -s $source_path --"
+    echo "LD=$docker_py cc --cc $container_cross_ld -i qemu/$container_image -s $source_path --"
+    echo "NM=$docker_py cc --cc $container_cross_nm -i qemu/$container_image -s $source_path --"
+    echo "OBJCOPY=$docker_py cc --cc $container_cross_objcopy -i qemu/$container_image -s $source_path --"
+    echo "RANLIB=$docker_py cc --cc $container_cross_ranlib -i qemu/$container_image -s $source_path --"
+    echo "STRIP=$docker_py cc --cc $container_cross_strip -i qemu/$container_image -s $source_path --"
+  else
+    if test -n "$target_cc"; then
+      echo "CC=$target_cc"
+      echo "CCAS=$target_ccas"
+    fi
+    if test -n "$target_ar"; then
+      echo "AR=$target_ar"
+    fi
+    if test -n "$target_as"; then
+      echo "AS=$target_as"
+    fi
+    if test -n "$target_ld"; then
+      echo "LD=$target_ld"
+    fi
+    if test -n "$target_nm"; then
+      echo "NM=$target_nm"
+    fi
+    if test -n "$target_objcopy"; then
+      echo "OBJCOPY=$target_objcopy"
+    fi
+    if test -n "$target_ranlib"; then
+      echo "RANLIB=$target_ranlib"
+    fi
+    if test -n "$target_strip"; then
+      echo "STRIP=$target_strip"
+    fi
+  fi
+}
+
 #######################################
 # cross-compiled firmware targets
 
@@ -1299,6 +1659,48 @@ fi
 
 echo "# Automatically generated by configure - do not modify" > Makefile.prereqs
 
+# Mac OS X ships with a broken assembler
+if have_target i386-softmmu x86_64-softmmu && \
+        test "$host_os" != "darwin" && test "$host_os" != "sunos" && \
+        test "$host_os" != "haiku" && \
+        probe_target_compiler i386-softmmu; then
+    subdirs="$subdirs pc-bios/optionrom"
+    config_mak=pc-bios/optionrom/config.mak
+    echo "# Automatically generated by configure - do not modify" > $config_mak
+    echo "TOPSRC_DIR=$source_path" >> $config_mak
+    write_target_makefile >> $config_mak
+fi
+
+if have_target ppc-softmmu ppc64-softmmu && \
+        probe_target_compiler ppc-softmmu; then
+    subdirs="$subdirs pc-bios/vof"
+    config_mak=pc-bios/vof/config.mak
+    echo "# Automatically generated by configure - do not modify" > $config_mak
+    echo "SRC_DIR=$source_path/pc-bios/vof" >> $config_mak
+    write_target_makefile >> $config_mak
+fi
+
+# Only build s390-ccw bios if the compiler has -march=z900 or -march=z10
+# (which is the lowest architecture level that Clang supports)
+if have_target s390x-softmmu && probe_target_compiler s390x-softmmu && \
+    GIT=git "$source_path/scripts/git-submodule.sh" "$git_submodules_action" roms/SLOF >> config.log 2>&1; then
+  write_c_skeleton
+  do_compiler "$target_cc" $target_cc_cflags -march=z900 -o $TMPO -c $TMPC
+  has_z900=$?
+  if [ $has_z900 = 0 ] || do_compiler "$target_cc" $target_cc_cflags -march=z10 -msoft-float -Werror -o $TMPO -c $TMPC; then
+    if [ $has_z900 != 0 ]; then
+      echo "WARNING: Your compiler does not support the z900!"
+      echo "         The s390-ccw bios will only work with guest CPUs >= z10."
+    fi
+    subdirs="$subdirs pc-bios/s390-ccw"
+    config_mak=pc-bios/s390-ccw/config-host.mak
+    echo "# Automatically generated by configure - do not modify" > $config_mak
+    echo "SRC_PATH=$source_path/pc-bios/s390-ccw" >> $config_mak
+    echo "GIT_SUBMODULES_ACTION=$git_submodules_action" >> $config_mak
+    write_target_makefile >> $config_mak
+  fi
+fi
+
 #######################################
 # generate config-host.mak
 
@@ -1322,6 +1724,65 @@ if test "$default_targets" = "yes"; then
   echo "CONFIG_DEFAULT_TARGETS=y" >> $config_host_mak
 fi
 
+tcg_tests_with_compilers=
+for target in $target_list; do
+  arch=${target%%-*}
+
+  case $target in
+    xtensa*-linux-user)
+      # the toolchain is not complete with headers, only build system tests
+      continue
+      ;;
+    *-softmmu)
+      test -f "$source_path/tests/tcg/$arch/Makefile.softmmu-target" || continue
+      qemu="qemu-system-$arch"
+      ;;
+    *-linux-user|*-bsd-user)
+      qemu="qemu-$arch"
+      ;;
+  esac
+
+  if probe_target_compiler $target || test -n "$container_image"; then
+      test -n "$container_image" && build_static=y
+      mkdir -p "tests/tcg/$target"
+      config_target_mak=tests/tcg/$target/config-target.mak
+      ln -sf "$source_path/tests/tcg/Makefile.target" "tests/tcg/$target/Makefile"
+      echo "# Automatically generated by configure - do not modify" > "$config_target_mak"
+      echo "TARGET_NAME=$arch" >> "$config_target_mak"
+      echo "TARGET=$target" >> "$config_target_mak"
+      write_target_makefile "build-tcg-tests-$target" >> "$config_target_mak"
+      echo "BUILD_STATIC=$build_static" >> "$config_target_mak"
+      echo "QEMU=$PWD/$qemu" >> "$config_target_mak"
+
+      # will GDB work with these binaries?
+      if test "${gdb_arches#*$arch}" != "$gdb_arches"; then
+          echo "GDB=$gdb_bin" >> $config_target_mak
+      fi
+
+      if test "${gdb_arches#*$arch}" != "$gdb_arches" && version_ge $gdb_version 14.1; then
+          echo "GDB_HAS_SME_TILES=y" >> $config_target_mak
+      else
+          echo "GDB_HAS_SME_TILES=n" >> $config_target_mak
+      fi
+
+      if test "${gdb_arches#*aarch64}" != "$gdb_arches" && version_ge $gdb_version 15.1; then
+          echo "GDB_HAS_MTE=y" >> $config_target_mak
+      fi
+
+      if test "${gdb_arches#*aarch64}" != "$gdb_arches" && version_ge $gdb_version 16.0; then
+          # GDB has to support MTE in baremetal to allow debugging MTE in QEMU system mode
+          echo "GDB_SUPPORTS_MTE_IN_BAREMETAL=y" >> $config_target_mak
+      fi
+
+      echo "run-tcg-tests-$target: $qemu\$(EXESUF)" >> Makefile.prereqs
+      tcg_tests_with_compilers="$tcg_tests_with_compilers $target"
+  fi
+done
+
+if test "$tcg" = "enabled"; then
+    echo "TCG_TESTS_WITH_COMPILERS=$tcg_tests_with_compilers" >> $config_host_mak
+fi
+
 if test "$skip_meson" = no; then
   cross="config-meson.cross.new"
   meson_quote() {
-- 
2.47.3

[PATCH v5 02/17] configure: ensure it also honours container configuration

[Alex Bennée]

This partially reverts the bits of 522b299ca43 (tests/tcg/meson.build:
implement ./configure containers options) so the configure script can
also honour the container config while keeping the plumbing to control
containers within meson.

Tested-by: Richard Henderson <richard.henderson@linaro.org>
Tested-by: Eric Farman <farman@linux.ibm.com>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
[AJB: untangle changes since]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Jared Rossi <jrossi@linux.ibm.com>

---
v5
  - reword and rename commit (we haven't actually reverted much)
  - rebase conflict
  - drop stray options
  - move registry bits
v4
  - tweak revert to keep disable reported
  - update commit msg
---
 configure | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/configure b/configure
index 1973f023b1e..8f8fec4efc8 100755
--- a/configure
+++ b/configure
@@ -246,6 +246,7 @@ ninja=""
 python=
 download="enabled"
 skip_meson=no
+use_containers="yes"
 rust="disabled"
 rust_target_triple=""
 
@@ -722,11 +723,14 @@ for opt do
   ;;
   --disable-plugins) plugins="no"
   ;;
-  --enable-containers) # enabled by default
+  --enable-containers) use_containers="yes"
+                       meson_option_add -Dcontainers=true
   ;;
-  --disable-containers) meson_option_add -Dcontainers=false
+  --disable-containers) use_containers="no"
+                        meson_option_add -Dcontainers=false
   ;;
-  --container-command=*) meson_option_add -Dcontainer_command="$optarg"
+  --container-command=*) container_command="$optarg"
+                         meson_option_add -Dcontainer_command="$optarg"
   ;;
   --rust-target-triple=*) rust_target_triple="$optarg"
   ;;
@@ -1713,6 +1717,9 @@ echo all: >> $config_host_mak
 
 echo "SRC_PATH=$source_path" >> $config_host_mak
 echo "TARGET_DIRS=$target_list" >> $config_host_mak
+if test "$container_command" != ""; then
+    echo "CONTAINER_COMMAND=$container_command" >> $config_host_mak
+fi
 echo "SUBDIRS=$subdirs" >> $config_host_mak
 echo "PYTHON=$python" >> $config_host_mak
 echo "MKVENV_ENSUREGROUP=$mkvenv ensuregroup $mkvenv_online_flag" >> $config_host_mak
-- 
2.47.3

[PATCH v5 03/17] configure: correctly honour --disable-containers

[Alex Bennée]

From: Daniel P. Berrangé <berrange@redhat.com>

The configure script originally only probed for 'runc' command
in order to handle cross-compilation containers. The Makefile
under tests/docker would then probe again when used.

In order to eliminate the additional probeing, we changed
configure in commit c4ce04cfb7460d46c0262d437a7f35bb5c5fc449
to always probe for 'runc', so the result could be passed
along to tests/docker make rules.

That commit overlooked that the 'probe_target_compiler' func
was relying on '$runc' == 'no' as a proxy for the state of
the --disable-containers arg.

When we started unconditionally probing, that short cut no
longer works and we must explicitly check "$use_containers"
too.

Fixes: c4ce04cfb7460d46c0262d437a7f35bb5c5fc449
Reported-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 configure | 1 +
 1 file changed, 1 insertion(+)

diff --git a/configure b/configure
index 8f8fec4efc8..04e312a111a 100755
--- a/configure
+++ b/configure
@@ -1397,6 +1397,7 @@ probe_target_compiler() {
   esac
 
   for host in $container_hosts; do
+    test "$use_containers" = "yes" || continue
     test "$container_command" != "" || continue
     test "$host" = "$cpu" || continue
     case $target_arch in
-- 
2.47.3

[PATCH v5 04/17] configure: re-purpose the tcg test compiler configs for vdso building

[Alex Bennée]

All the tcg tests are in meson now so there is no reason to put the
config-target.mak files in those directories. We also don't need the
GDB feature tests or to emit prerequsistes.

The vdso re-build blob does use the compiler definitions though so
we can emit the details for that and tweak the vdso makefiles to point
at the right place. Note some of the vdso makefiles build both 32 and
64 bit versions from the same compiler.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 configure                            | 69 +++++++++++-----------------
 linux-user/aarch64/Makefile.vdso     |  2 +-
 linux-user/arm/Makefile.vdso         |  2 +-
 linux-user/hppa/Makefile.vdso        |  2 +-
 linux-user/i386/Makefile.vdso        |  2 +-
 linux-user/loongarch64/Makefile.vdso |  2 +-
 linux-user/ppc/Makefile.vdso         |  2 +-
 linux-user/riscv/Makefile.vdso       |  2 +-
 linux-user/s390x/Makefile.vdso       |  2 +-
 linux-user/sh4/Makefile.vdso         |  2 +-
 linux-user/x86_64/Makefile.vdso      |  2 +-
 11 files changed, 36 insertions(+), 53 deletions(-)

diff --git a/configure b/configure
index 04e312a111a..667fc7ca19b 100755
--- a/configure
+++ b/configure
@@ -1732,65 +1732,48 @@ if test "$default_targets" = "yes"; then
   echo "CONFIG_DEFAULT_TARGETS=y" >> $config_host_mak
 fi
 
-tcg_tests_with_compilers=
+# Export the compiler details needed for rebuilding the vdso blobs
 for target in $target_list; do
-  arch=${target%%-*}
 
+  # Skip anything that is not a linux-user target
   case $target in
+    *-linux-user)
+      ;;
+    *)
+      continue
+      ;;
+  esac
+
+  arch=${target%%-*}
+
+  case $arch in
     xtensa*-linux-user)
-      # the toolchain is not complete with headers, only build system tests
+      # the toolchain can only build system tests due to missing headers
+      continue
+      ;;
+    aarch64_be-linux-user)
+      # there is no BE user space
       continue
       ;;
-    *-softmmu)
-      test -f "$source_path/tests/tcg/$arch/Makefile.softmmu-target" || continue
-      qemu="qemu-system-$arch"
+    aarch64|arm|hppa|i386|loongarch64|ppc64|riscv64|s390x|sh4|x86_64)
+      # arches which need a vdso, some 64 bit arches build both
       ;;
-    *-linux-user|*-bsd-user)
-      qemu="qemu-$arch"
+    *)
+      continue
       ;;
   esac
 
-  if probe_target_compiler $target || test -n "$container_image"; then
-      test -n "$container_image" && build_static=y
-      mkdir -p "tests/tcg/$target"
-      config_target_mak=tests/tcg/$target/config-target.mak
-      ln -sf "$source_path/tests/tcg/Makefile.target" "tests/tcg/$target/Makefile"
+  # note container_image is set by a successful call to probe_target_compiler
+  if probe_target_compiler "$target" || test -n "$container_image"; then
+      mkdir -p "linux-user/$arch"
+      config_target_mak=linux-user/$arch/config-target.mak
       echo "# Automatically generated by configure - do not modify" > "$config_target_mak"
       echo "TARGET_NAME=$arch" >> "$config_target_mak"
       echo "TARGET=$target" >> "$config_target_mak"
-      write_target_makefile "build-tcg-tests-$target" >> "$config_target_mak"
-      echo "BUILD_STATIC=$build_static" >> "$config_target_mak"
-      echo "QEMU=$PWD/$qemu" >> "$config_target_mak"
-
-      # will GDB work with these binaries?
-      if test "${gdb_arches#*$arch}" != "$gdb_arches"; then
-          echo "GDB=$gdb_bin" >> $config_target_mak
-      fi
-
-      if test "${gdb_arches#*$arch}" != "$gdb_arches" && version_ge $gdb_version 14.1; then
-          echo "GDB_HAS_SME_TILES=y" >> $config_target_mak
-      else
-          echo "GDB_HAS_SME_TILES=n" >> $config_target_mak
-      fi
-
-      if test "${gdb_arches#*aarch64}" != "$gdb_arches" && version_ge $gdb_version 15.1; then
-          echo "GDB_HAS_MTE=y" >> $config_target_mak
-      fi
-
-      if test "${gdb_arches#*aarch64}" != "$gdb_arches" && version_ge $gdb_version 16.0; then
-          # GDB has to support MTE in baremetal to allow debugging MTE in QEMU system mode
-          echo "GDB_SUPPORTS_MTE_IN_BAREMETAL=y" >> $config_target_mak
-      fi
-
-      echo "run-tcg-tests-$target: $qemu\$(EXESUF)" >> Makefile.prereqs
-      tcg_tests_with_compilers="$tcg_tests_with_compilers $target"
+      write_target_makefile "update-linux-vdso" >> "$config_target_mak"
   fi
 done
 
-if test "$tcg" = "enabled"; then
-    echo "TCG_TESTS_WITH_COMPILERS=$tcg_tests_with_compilers" >> $config_host_mak
-fi
-
 if test "$skip_meson" = no; then
   cross="config-meson.cross.new"
   meson_quote() {
diff --git a/linux-user/aarch64/Makefile.vdso b/linux-user/aarch64/Makefile.vdso
index c33a679c0f2..82ed2011087 100644
--- a/linux-user/aarch64/Makefile.vdso
+++ b/linux-user/aarch64/Makefile.vdso
@@ -1,4 +1,4 @@
-include $(BUILD_DIR)/tests/tcg/aarch64-linux-user/config-target.mak
+include $(BUILD_DIR)/linux-user/aarch64/config-target.mak
 
 SUBDIR = $(SRC_PATH)/linux-user/aarch64
 VPATH += $(SUBDIR)
diff --git a/linux-user/arm/Makefile.vdso b/linux-user/arm/Makefile.vdso
index ede489e2363..16ae8d7aaae 100644
--- a/linux-user/arm/Makefile.vdso
+++ b/linux-user/arm/Makefile.vdso
@@ -1,4 +1,4 @@
-include $(BUILD_DIR)/tests/tcg/arm-linux-user/config-target.mak
+include $(BUILD_DIR)/linux-user/arm/config-target.mak
 
 SUBDIR = $(SRC_PATH)/linux-user/arm
 VPATH += $(SUBDIR)
diff --git a/linux-user/hppa/Makefile.vdso b/linux-user/hppa/Makefile.vdso
index f4537ae7169..78642c42661 100644
--- a/linux-user/hppa/Makefile.vdso
+++ b/linux-user/hppa/Makefile.vdso
@@ -1,4 +1,4 @@
-include $(BUILD_DIR)/tests/tcg/hppa-linux-user/config-target.mak
+include $(BUILD_DIR)/linux-user/hppa/config-target.mak
 
 SUBDIR = $(SRC_PATH)/linux-user/hppa
 VPATH += $(SUBDIR)
diff --git a/linux-user/i386/Makefile.vdso b/linux-user/i386/Makefile.vdso
index 95bc616f6d6..ca2e37fa6b6 100644
--- a/linux-user/i386/Makefile.vdso
+++ b/linux-user/i386/Makefile.vdso
@@ -1,4 +1,4 @@
-include $(BUILD_DIR)/tests/tcg/i386-linux-user/config-target.mak
+include $(BUILD_DIR)/linux-user/i386/config-target.mak
 
 SUBDIR = $(SRC_PATH)/linux-user/i386
 VPATH += $(SUBDIR)
diff --git a/linux-user/loongarch64/Makefile.vdso b/linux-user/loongarch64/Makefile.vdso
index 1d760b1e47e..ceb61101209 100644
--- a/linux-user/loongarch64/Makefile.vdso
+++ b/linux-user/loongarch64/Makefile.vdso
@@ -1,4 +1,4 @@
-include $(BUILD_DIR)/tests/tcg/loongarch64-linux-user/config-target.mak
+include $(BUILD_DIR)/linux-user/loongarch64/config-target.mak
 
 SUBDIR = $(SRC_PATH)/linux-user/loongarch64
 VPATH += $(SUBDIR)
diff --git a/linux-user/ppc/Makefile.vdso b/linux-user/ppc/Makefile.vdso
index e2b8facbb50..6b9be843817 100644
--- a/linux-user/ppc/Makefile.vdso
+++ b/linux-user/ppc/Makefile.vdso
@@ -1,4 +1,4 @@
-include $(BUILD_DIR)/tests/tcg/ppc64-linux-user/config-target.mak
+include $(BUILD_DIR)/linux-user/ppc64/config-target.mak
 
 SUBDIR = $(SRC_PATH)/linux-user/ppc
 VPATH += $(SUBDIR)
diff --git a/linux-user/riscv/Makefile.vdso b/linux-user/riscv/Makefile.vdso
index 2c257dbfda5..2eefdb70cf9 100644
--- a/linux-user/riscv/Makefile.vdso
+++ b/linux-user/riscv/Makefile.vdso
@@ -1,4 +1,4 @@
-include $(BUILD_DIR)/tests/tcg/riscv64-linux-user/config-target.mak
+include $(BUILD_DIR)/linux-user/riscv64/config-target.mak
 
 SUBDIR = $(SRC_PATH)/linux-user/riscv
 VPATH += $(SUBDIR)
diff --git a/linux-user/s390x/Makefile.vdso b/linux-user/s390x/Makefile.vdso
index e82bf9e29ff..51f7c50a2de 100644
--- a/linux-user/s390x/Makefile.vdso
+++ b/linux-user/s390x/Makefile.vdso
@@ -1,4 +1,4 @@
-include $(BUILD_DIR)/tests/tcg/s390x-linux-user/config-target.mak
+include $(BUILD_DIR)/linux-user/s390x/config-target.mak
 
 SUBDIR = $(SRC_PATH)/linux-user/s390x
 VPATH += $(SUBDIR)
diff --git a/linux-user/sh4/Makefile.vdso b/linux-user/sh4/Makefile.vdso
index 272c5d262de..ed2cc11b825 100644
--- a/linux-user/sh4/Makefile.vdso
+++ b/linux-user/sh4/Makefile.vdso
@@ -1,4 +1,4 @@
-include $(BUILD_DIR)/tests/tcg/sh4-linux-user/config-target.mak
+include $(BUILD_DIR)/linux-user/sh4/config-target.mak
 
 SUBDIR = $(SRC_PATH)/linux-user/sh4
 VPATH += $(SUBDIR)
diff --git a/linux-user/x86_64/Makefile.vdso b/linux-user/x86_64/Makefile.vdso
index 26552b66dba..56fe87af72b 100644
--- a/linux-user/x86_64/Makefile.vdso
+++ b/linux-user/x86_64/Makefile.vdso
@@ -1,4 +1,4 @@
-include $(BUILD_DIR)/tests/tcg/x86_64-linux-user/config-target.mak
+include $(BUILD_DIR)/linux-user/x86_64/config-target.mak
 
 SUBDIR = $(SRC_PATH)/linux-user/x86_64
 VPATH += $(SUBDIR)
-- 
2.47.3

[PATCH v5 05/17] tests/tcg/s390x: Provide the libc bits needed by the system tests

[Alex Bennée]

From: Ilya Leoshkevich <iii@linux.ibm.com>

The meson conversion dropped the SLOF libc include path that commit
3fa010d5317 ("tests/tcg/s390x: Use the SLOF libc headers for the
multiarch tests") had added, breaking the build on systems without a
cross-libc, such as Fedora.

Rather than reintroducing a dependency on the SLOF submodule, which no
other tcg test has, declare the three functions that the reused pc-bios
code actually needs: memcpy(), memset(), and printf().

Reported-by: Daniel P. Berrangé <berrange@redhat.com>
Reported-by: Alex Bennée <alex.bennee@linaro.org>
Closes: https://patchew.org/QEMU/20260910100120.2076092-1-berrange@redhat.com/
Suggested-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Fixes: cd81cf86834 ("tests/tcg/s390x: system tests")
Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Message-ID: <20260915110959.492195-1-iii@linux.ibm.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/tcg/s390x/libc/stdio.h       | 11 +++++++++++
 tests/tcg/s390x/libc/string.h      | 12 ++++++++++++
 tests/tcg/s390x/system/meson.build |  2 ++
 3 files changed, 25 insertions(+)
 create mode 100644 tests/tcg/s390x/libc/stdio.h
 create mode 100644 tests/tcg/s390x/libc/string.h

diff --git a/tests/tcg/s390x/libc/stdio.h b/tests/tcg/s390x/libc/stdio.h
new file mode 100644
index 00000000000..c3e45995f65
--- /dev/null
+++ b/tests/tcg/s390x/libc/stdio.h
@@ -0,0 +1,11 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+#ifndef QEMU_TESTS_S390X_STDIO_H
+#define QEMU_TESTS_S390X_STDIO_H
+
+#include <minilib.h>
+
+#define printf ml_printf
+
+#endif
diff --git a/tests/tcg/s390x/libc/string.h b/tests/tcg/s390x/libc/string.h
new file mode 100644
index 00000000000..1cabeb90adf
--- /dev/null
+++ b/tests/tcg/s390x/libc/string.h
@@ -0,0 +1,12 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+#ifndef QEMU_TESTS_S390X_STRING_H
+#define QEMU_TESTS_S390X_STRING_H
+
+#include <stddef.h>
+
+void *memcpy(void *dest, const void *src, size_t n);
+void *memset(void *dest, int c, size_t size);
+
+#endif
diff --git a/tests/tcg/s390x/system/meson.build b/tests/tcg/s390x/system/meson.build
index ca07f503058..1e360e70821 100644
--- a/tests/tcg/s390x/system/meson.build
+++ b/tests/tcg/s390x/system/meson.build
@@ -6,11 +6,13 @@ link_script = files('../softmmu.ld')[0]
 head64 = files('../head64.S')[0]
 console = files('../console.c')[0]
 ipl_inc = meson.project_source_root()/'include'/'hw'/'s390x'/'ipl'
+libc_inc = meson.current_source_dir()/'..'/'libc'
 cflags = ['-nostdlib',
           '-ffreestanding',
           '-Wa,--noexecstack',
           '-I', minilib_dir, minilib_printf,
           '-I', ipl_inc,
+          '-I', libc_inc,
           '-march=z13',
           head64, console]
 asmflags = ['-nostdlib',
-- 
2.47.3

[PATCH v5 06/17] tests/tcg: prefix meson built docker tags with qemu/

[Alex Bennée]

With the meson changes we now build the images directly by calling
docker.py. However the Makefile.include builds prefix all the images
with qemu/:

  # General rule for building docker images.
  docker-image-%: $(DOCKER_FILES_DIR)/%.docker
            $(call quiet-command,			\
                  DOCKER_BUILDKIT=1 $(CONTAINER_COMMAND) build \
                  $(if $(DOCKER_V),,--quiet)		\
                  $(if $(NOCACHE),--no-cache,		\
                          $(if $(DOCKER_REGISTRY),--cache-from $(DOCKER_REGISTRY)/qemu/$*)) \
                  --build-arg BUILDKIT_INLINE_CACHE=1 	\
                  $(if $(NOUSER),,			\
                          --build-arg USER=$(USER)	\
                          --build-arg UID=$(UID))		\
                  -t qemu/$* - < $< $(if $(DOCKER_V),,> /dev/null),\
                  "BUILD", $*)

So to ensure we don't end up building every container twice make sure
the naming conventions line up.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/tcg/meson.build | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/tcg/meson.build b/tests/tcg/meson.build
index d41a228fb3d..0604c4b90b9 100644
--- a/tests/tcg/meson.build
+++ b/tests/tcg/meson.build
@@ -255,7 +255,7 @@ foreach target, plan: tcg_tests
     image_name = 'image-' + cc_dockerfile
     has_docker = docker_host_arch_supported and docker_supported
     if cc_dockerfile not in image_targets and has_docker
-      cmd = [docker_wrapper, 'build', '-f', dockerfile, '-t', cc_dockerfile]
+      cmd = [docker_wrapper, 'build', '-f', dockerfile, '-t', 'qemu/' + cc_dockerfile]
       t = custom_target(image_name, command: cmd,
                         build_by_default: false,
                         output: 'no_output_' + image_name)
@@ -270,7 +270,7 @@ foreach target, plan: tcg_tests
       here = meson.project_build_root()
       cc = [docker_wrapper, 'run', '--run-as-current-user',
             '-w', here, '-v', mount,
-            cc_dockerfile, plan['cc']]
+            'qemu/' + cc_dockerfile, plan['cc']]
       has_cc = true
     endif
   endif
@@ -279,7 +279,7 @@ foreach target, plan: tcg_tests
   if has_cc
     summary_cc = plan['cc']
     if not cc_from_system
-      summary_cc += ' (from \'' + plan['cc_dockerfile'] + '\' container)'
+      summary_cc += ' (from \'qemu/' + plan['cc_dockerfile'] + '\' container)'
     endif
   endif
   tcg_tests_summary += {'cc for ' + target: summary_cc}
-- 
2.47.3

[PATCH v5 07/17] tests/tcg: use --add-current-user when building containers

[Alex Bennée]

As the meson tcg build routes the container builds through docker.py
we can also clean up its hack which involved appending a RUN line to
the dockerfile. All the dockerfiles that care will work with the
--build-arg.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 13 ++++++-------
 tests/tcg/meson.build  |  3 ++-
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 315fa2ff848..6090a9dc957 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -297,13 +297,6 @@ def build_image(self, tag, docker_dir, dockerfile,
                                              dir=docker_dir, suffix=".docker")
         tmp_df.write(dockerfile)
 
-        if user:
-            uid = os.getuid()
-            uname = getpass.getuser()
-            tmp_df.write("\n")
-            tmp_df.write("RUN id %s 2>/dev/null || useradd -u %d -U %s" %
-                         (uname, uid, uname))
-
         tmp_df.write("\n")
         tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s\n" % (checksum))
         for f, c in extra_files_cksum:
@@ -315,6 +308,12 @@ def build_image(self, tag, docker_dir, dockerfile,
         if self._buildkit:
             build_args += ["--build-arg", "BUILDKIT_INLINE_CACHE=1"]
 
+        if user:
+            uid = os.getuid()
+            uname = getpass.getuser()
+            build_args += ["--build-arg", "USER=%s" % uname,
+                           "--build-arg", "UID=%s" % uid]
+
         if registry is not None:
             pull_args = ["pull", "%s/%s" % (registry, tag)]
             self._do(pull_args, quiet=quiet)
diff --git a/tests/tcg/meson.build b/tests/tcg/meson.build
index 0604c4b90b9..814f2f8cd8c 100644
--- a/tests/tcg/meson.build
+++ b/tests/tcg/meson.build
@@ -255,7 +255,8 @@ foreach target, plan: tcg_tests
     image_name = 'image-' + cc_dockerfile
     has_docker = docker_host_arch_supported and docker_supported
     if cc_dockerfile not in image_targets and has_docker
-      cmd = [docker_wrapper, 'build', '-f', dockerfile, '-t', 'qemu/' + cc_dockerfile]
+      cmd = [docker_wrapper, 'build', '-f', dockerfile, '-t', 'qemu/' + cc_dockerfile,
+             '--add-current-user']
       t = custom_target(image_name, command: cmd,
                         build_by_default: false,
                         output: 'no_output_' + image_name)
-- 
2.47.3

[PATCH v5 08/17] tests/tcg: don't define tests when missing compilers

[Alex Bennée]

Skip defining the tests if we don't have a compiler to build them.
Otherwise we add a bunch of noise to the reported results. We still do
report when we skip because the compiler doesn't have the features we
need to build.

Suggested-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v5
  - alternate patch from Pierrick which keeps validation
---
 tests/tcg/meson.build | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/tests/tcg/meson.build b/tests/tcg/meson.build
index 814f2f8cd8c..5819fdf0bf3 100644
--- a/tests/tcg/meson.build
+++ b/tests/tcg/meson.build
@@ -499,10 +499,14 @@ foreach target, plan: tcg_tests
         cmd = ['sh', '-c', 'echo "' + skip_test + '" && exit 77']
       endif
 
-      test(test_name, env, args: cmd,
-           depends: test_depends,
-           timeout: 120,
-           suite: ['tcg', 'tcg-' + target])
+      # Do not add a skipped test when compiler is missing,
+      # though we still check all tests are correctly defined.
+      if has_cc
+        test(test_name, env, args: cmd,
+             depends: test_depends,
+             timeout: 120,
+             suite: ['tcg', 'tcg-' + target])
+      endif
       added_tests += {test_name: true}
     endforeach
   endforeach
-- 
2.47.3

[PATCH v5 09/17] tests/docker: add container registry to configure and use

[Alex Bennée]

Previously we had the default baked into Makefile.include but as the
meson container logic will also benefit from it we update the configure
machinery.

For now we allow direct make invocations to override but for the
normal case it is in one place now.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - fixed conflicts from the revert.
v3
  - regen meson-buildoptions.sh
v4
  - merge conflicts
v5
  - fix strays on rebase
  - update meson_options.txt
---
 configure                     | 7 +++++++
 meson_options.txt             | 3 +++
 scripts/meson-buildoptions.sh | 3 +++
 tests/docker/Makefile.include | 4 +---
 tests/tcg/meson.build         | 6 ++++++
 5 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/configure b/configure
index 667fc7ca19b..af14a9c23ef 100755
--- a/configure
+++ b/configure
@@ -172,6 +172,7 @@ fi
 # some defaults, based on the host environment
 
 # default parameters
+container_registry="registry.gitlab.com/qemu-project/qemu"
 cpu=""
 cross_compile="no"
 cross_prefix=""
@@ -732,6 +733,9 @@ for opt do
   --container-command=*) container_command="$optarg"
                          meson_option_add -Dcontainer_command="$optarg"
   ;;
+  --container-registry=*) container_registry="$optarg"
+                          meson_option_add -Dcontainer_registry="$optarg"
+  ;;
   --rust-target-triple=*) rust_target_triple="$optarg"
   ;;
   --gdb=*) meson_option_add -Dgdb="$optarg"
@@ -1728,6 +1732,9 @@ echo "GENISOIMAGE=$genisoimage" >> $config_host_mak
 echo "MESON=$meson" >> $config_host_mak
 echo "NINJA=$ninja" >> $config_host_mak
 echo "EXESUF=$EXESUF" >> $config_host_mak
+if test -n "$container_registry"; then
+  echo "DOCKER_DEFAULT_REGISTRY=$container_registry" >> $config_host_mak
+fi
 if test "$default_targets" = "yes"; then
   echo "CONFIG_DEFAULT_TARGETS=y" >> $config_host_mak
 fi
diff --git a/meson_options.txt b/meson_options.txt
index e3570a418aa..6c2de6296f7 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -393,6 +393,9 @@ option('containers', type: 'boolean', value: true,
        description: 'use containers to cross compile tcg tests')
 option('container_command', type: 'string',
        description: 'command to build/run containers')
+option('container_registry', type: 'string',
+       value: 'registry.gitlab.com/qemu-project/qemu',
+       description: 'container registry to cache from')
 
 option('tcg_tests_cross_cc_aarch64', type: 'string',
        description: 'cc for aarch64 tcg tests')
diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
index ba3c9de216b..0cb1d48c982 100644
--- a/scripts/meson-buildoptions.sh
+++ b/scripts/meson-buildoptions.sh
@@ -10,6 +10,9 @@ meson_options_help() {
   printf "%s\n" '  --block-drv-rw-whitelist=VALUE'
   printf "%s\n" '                           set block driver read-write whitelist (by default'
   printf "%s\n" '                           affects only QEMU, not tools like qemu-img)'
+  printf "%s\n" '  --container-registry=VALUE'
+  printf "%s\n" '                           container registry to cache from'
+  printf "%s\n" '                           [registry.gitlab.com/qemu-project/qemu]'
   printf "%s\n" '  --datadir=VALUE          Data file directory [share]'
   printf "%s\n" '  --disable-coroutine-pool coroutine freelist (better performance)'
   printf "%s\n" '  --disable-debug-info     Enable debug symbols and other information'
diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 0adddb6a5c6..74bf471f171 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -11,9 +11,7 @@ USER = $(if $(NOUSER),,$(shell id -un))
 UID = $(if $(NOUSER),,$(shell id -u))
 
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
-ifeq ($(HOST_ARCH),x86_64)
-DOCKER_DEFAULT_REGISTRY := registry.gitlab.com/qemu-project/qemu
-endif
+DOCKER_DEFAULT_REGISTRY ?= registry.gitlab.com/qemu-project/qemu
 DOCKER_REGISTRY := $(if $(REGISTRY),$(REGISTRY),$(DOCKER_DEFAULT_REGISTRY))
 
 CONTAINER_COMMAND ?= $(shell $(SRC_PATH)/tests/docker/docker.py probe)
diff --git a/tests/tcg/meson.build b/tests/tcg/meson.build
index 5819fdf0bf3..482de73ac72 100644
--- a/tests/tcg/meson.build
+++ b/tests/tcg/meson.build
@@ -101,6 +101,9 @@ if get_option('containers')
   endif
 endif
 tcg_tests_summary += {'container command': container_command}
+if get_option('container_registry') != ''
+  tcg_tests_summary += {'container registry': get_option('container_registry')}
+endif
 
 # plugins come first, as we need to build the list
 test_plugins = {}
@@ -257,6 +260,9 @@ foreach target, plan: tcg_tests
     if cc_dockerfile not in image_targets and has_docker
       cmd = [docker_wrapper, 'build', '-f', dockerfile, '-t', 'qemu/' + cc_dockerfile,
              '--add-current-user']
+      if get_option('container_registry') != ''
+        cmd += ['--registry', get_option('container_registry')]
+      endif
       t = custom_target(image_name, command: cmd,
                         build_by_default: false,
                         output: 'no_output_' + image_name)
-- 
2.47.3

[PATCH v5 10/17] tests/docker: remove used _read_qemu_dockerfile

[Alex Bennée]

This is no longer needed since the functionality using it was removed.

Fixes: 07056db1b50 (tests/docker: remove FROM qemu/ support from docker.py)
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 10 ----------
 1 file changed, 10 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 6090a9dc957..fd0cbdfe856 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -175,16 +175,6 @@ def _check_binfmt_misc(executable):
     return interp, True
 
 
-def _read_qemu_dockerfile(img_name):
-    # special case for Debian linux-user images
-    if img_name.startswith("debian") and img_name.endswith("user"):
-        img_name = "debian-bootstrap"
-
-    df = os.path.join(os.path.dirname(__file__), "dockerfiles",
-                      img_name + ".docker")
-    return _read_dockerfile(df)
-
-
 def _dockerfile_verify_flat(df):
     "Verify we do not include other qemu/ layers"
     for l in df.splitlines():
-- 
2.47.3

[PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Alex Bennée]

The caching logic was a remnant of an older time when we had multiple
shared layers building that went into the final image. It was arguable
that it worked back then but it is definitely not needed now so into
the bin it goes.

Tested-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/docker/docker.py | 148 ++++++++++++++---------------------------
 1 file changed, 49 insertions(+), 99 deletions(-)

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index fd0cbdfe856..a2e97061240 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -15,7 +15,6 @@
 import sys
 import subprocess
 import json
-import hashlib
 import atexit
 import uuid
 import argparse
@@ -27,7 +26,6 @@
 from tarfile import TarFile, TarInfo
 from io import StringIO, BytesIO
 from shutil import copy, rmtree
-from datetime import datetime, timedelta
 
 
 FILTERED_ENV_NAMES = ['ftp_proxy', 'http_proxy', 'https_proxy']
@@ -35,20 +33,9 @@
 
 DEVNULL = open(os.devnull, 'wb')
 
-def _bytes_checksum(bytes):
-    """Calculate a digest string unique to the text content"""
-    return hashlib.sha1(bytes).hexdigest()
-
-def _text_checksum(text):
-    """Calculate a digest string unique to the text content"""
-    return _bytes_checksum(text.encode('utf-8'))
-
 def _read_dockerfile(path):
     return open(path, 'rt', encoding='utf-8').read()
 
-def _file_checksum(filename):
-    return _bytes_checksum(open(filename, 'rb').read())
-
 
 def _guess_engine_command():
     """ Guess a working engine command or raise exception if not found"""
@@ -263,38 +250,15 @@ def inspect_tag(self, tag):
         except subprocess.CalledProcessError:
             return None
 
-    def get_image_creation_time(self, info):
-        return json.loads(info)[0]["Created"]
-
-    def get_image_dockerfile_checksum(self, tag):
-        resp = self.inspect_tag(tag)
-        labels = json.loads(resp)[0]["Config"].get("Labels", {})
-        return labels.get("com.qemu.dockerfile-checksum", "")
-
-    def build_image(self, tag, docker_dir, dockerfile,
-                    quiet=True, user=False, argv=None, registry=None,
-                    extra_files_cksum=[]):
+    def build_image(self, tag, docker_dir, dockerfile_path,
+                    quiet=True, user=False, argv=None, registry=None):
         if argv is None:
             argv = []
 
-        if not _dockerfile_verify_flat(dockerfile):
+        if not _dockerfile_verify_flat(_read_dockerfile(dockerfile_path)):
             return -1
 
-        checksum = _text_checksum(dockerfile)
-
-        tmp_df = tempfile.NamedTemporaryFile(mode="w+t",
-                                             encoding='utf-8',
-                                             dir=docker_dir, suffix=".docker")
-        tmp_df.write(dockerfile)
-
-        tmp_df.write("\n")
-        tmp_df.write("LABEL com.qemu.dockerfile-checksum=%s\n" % (checksum))
-        for f, c in extra_files_cksum:
-            tmp_df.write("LABEL com.qemu.%s-checksum=%s\n" % (f, c))
-
-        tmp_df.flush()
-
-        build_args = ["build", "-t", tag, "-f", tmp_df.name]
+        build_args = ["build", "-t", tag, "-f", dockerfile_path]
         if self._buildkit:
             build_args += ["--build-arg", "BUILDKIT_INLINE_CACHE=1"]
 
@@ -320,13 +284,6 @@ def update_image(self, tag, tarball, quiet=True):
 
         self._do_check(["build", "-t", tag, "-"], quiet=quiet, stdin=tarball)
 
-    def image_matches_dockerfile(self, tag, dockerfile):
-        try:
-            checksum = self.get_image_dockerfile_checksum(tag)
-        except Exception:
-            return False
-        return checksum == _text_checksum(dockerfile)
-
     def run(self, cmd, keep, quiet, as_user=False):
         label = uuid.uuid4().hex
         if not keep:
@@ -412,61 +369,54 @@ def args(self, parser):
                             help="Dockerfile name")
 
     def run(self, args, argv):
-        dockerfile = _read_dockerfile(args.dockerfile)
         tag = args.tag
 
         dkr = Docker(args.command)
-        if "--no-cache" not in argv and \
-           dkr.image_matches_dockerfile(tag, dockerfile):
-            pass
-        else:
-            # Create a docker context directory for the build
-            docker_dir = tempfile.mkdtemp(prefix="docker_build")
-
-            # Validate binfmt_misc will work
-            if args.skip_binfmt:
-                qpath = args.include_executable
-            elif args.include_executable:
-                qpath, enabled = _check_binfmt_misc(args.include_executable)
-                if not enabled:
-                    return 1
-
-            # Is there a .pre file to run in the build context?
-            docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
-            if os.path.exists(docker_pre):
-                stdout = DEVNULL if args.quiet else None
-                rc = subprocess.call(os.path.realpath(docker_pre),
-                                     cwd=docker_dir, stdout=stdout)
-                if rc == 3:
-                    print("Skip")
-                    return 0
-                elif rc != 0:
-                    print("%s exited with code %d" % (docker_pre, rc))
-                    return 1
-
-            # Copy any extra files into the Docker context. These can be
-            # included by the use of the ADD directive in the Dockerfile.
-            cksum = []
-            if args.include_executable:
-                # FIXME: there is no checksum of this executable and the linked
-                # libraries, once the image built any change of this executable
-                # or any library won't trigger another build.
-                _copy_binary_with_libs(args.include_executable,
-                                       qpath, docker_dir)
-
-            for filename in args.extra_files or []:
-                _copy_with_mkdir(filename, docker_dir)
-                cksum += [(filename, _file_checksum(filename))]
-
-            argv += ["--build-arg=" + k.lower() + "=" + v
-                     for k, v in os.environ.items()
-                     if k.lower() in FILTERED_ENV_NAMES]
-            dkr.build_image(tag, docker_dir, dockerfile,
-                            quiet=args.quiet, user=args.user,
-                            argv=argv, registry=args.registry,
-                            extra_files_cksum=cksum)
-
-            rmtree(docker_dir)
+
+        # Create a docker context directory for the build
+        docker_dir = tempfile.mkdtemp(prefix="docker_build")
+
+        # Validate binfmt_misc will work
+        if args.skip_binfmt:
+            qpath = args.include_executable
+        elif args.include_executable:
+            qpath, enabled = _check_binfmt_misc(args.include_executable)
+            if not enabled:
+                return 1
+
+        # Is there a .pre file to run in the build context?
+        docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
+        if os.path.exists(docker_pre):
+            stdout = DEVNULL if args.quiet else None
+            rc = subprocess.call(os.path.realpath(docker_pre),
+                                 cwd=docker_dir, stdout=stdout)
+            if rc == 3:
+                print("Skip")
+                return 0
+            elif rc != 0:
+                print("%s exited with code %d" % (docker_pre, rc))
+                return 1
+
+        # Copy any extra files into the Docker context. These can be
+        # included by the use of the ADD directive in the Dockerfile.
+        if args.include_executable:
+            # FIXME: there is no checksum of this executable and the linked
+            # libraries, once the image built any change of this executable
+            # or any library won't trigger another build.
+            _copy_binary_with_libs(args.include_executable,
+                                   qpath, docker_dir)
+
+        for filename in args.extra_files or []:
+            _copy_with_mkdir(filename, docker_dir)
+
+        argv += ["--build-arg=" + k.lower() + "=" + v
+                 for k, v in os.environ.items()
+                 if k.lower() in FILTERED_ENV_NAMES]
+        dkr.build_image(tag, docker_dir, args.dockerfile,
+                        quiet=args.quiet, user=args.user,
+                        argv=argv, registry=args.registry)
+
+        rmtree(docker_dir)
 
         return 0
 
-- 
2.47.3

[PATCH v5 12/17] tests/docker: add specific command to parser

[Alex Bennée]

A future change will want to use additional flags which clash with
docker/podman command lines. Currently they would get snarfed up by
argparser. Make the pass-through commands explicit to avoid this.

Tested-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 scripts/coverity-scan/run-coverity-scan | 2 +-
 tests/docker/docker.py                  | 5 ++++-
 tests/tcg/meson.build                   | 2 +-
 3 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/scripts/coverity-scan/run-coverity-scan b/scripts/coverity-scan/run-coverity-scan
index 2f9ccd5e44b..0e1361606e0 100755
--- a/scripts/coverity-scan/run-coverity-scan
+++ b/scripts/coverity-scan/run-coverity-scan
@@ -381,7 +381,7 @@ if [ "$DOCKER" = yes ]; then
     # Arrange for this docker run to get access to the sources with -v.
     # We pass through all the configuration from the outer script to the inner.
     export COVERITY_EMAIL COVERITY_BUILD_CMD
-    tests/docker/docker.py run -it --env COVERITY_EMAIL --env COVERITY_BUILD_CMD \
+    tests/docker/docker.py run -- -it --env COVERITY_EMAIL --env COVERITY_BUILD_CMD \
            -v "$SECRETDIR:/work" coverity-scanner \
            ./run-coverity-scan --version "$VERSION" \
            --description "$DESCRIPTION" $ARGS --tokenfile /work/token \
diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index a2e97061240..d7eb4cf6811 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -336,9 +336,12 @@ def args(self, parser):
                             help="Don't remove image when command completes")
         parser.add_argument("--run-as-current-user", action="store_true",
                             help="Run container using the current user's uid")
+        parser.add_argument('cmd', nargs='*',
+                            help="""The command to run. You should precede with
+                            -- to avoid confusion about its flags""")
 
     def run(self, args, argv):
-        return Docker(args.command).run(argv, args.keep, quiet=args.quiet,
+        return Docker(args.command).run(args.cmd, args.keep, quiet=args.quiet,
                                         as_user=args.run_as_current_user)
 
 
diff --git a/tests/tcg/meson.build b/tests/tcg/meson.build
index 482de73ac72..053ef147df0 100644
--- a/tests/tcg/meson.build
+++ b/tests/tcg/meson.build
@@ -275,7 +275,7 @@ foreach target, plan: tcg_tests
       mount = meson.project_source_root()
       mount = mount + ':' + mount
       here = meson.project_build_root()
-      cc = [docker_wrapper, 'run', '--run-as-current-user',
+      cc = [docker_wrapper, 'run', '--run-as-current-user', '--',
             '-w', here, '-v', mount,
             'qemu/' + cc_dockerfile, plan['cc']]
       has_cc = true
-- 
2.47.3

[PATCH v5 13/17] tests/docker: fix ccache directory variable expansion

[Alex Bennée]

From: Anders Roxell <anders.roxell@linaro.org>

DOCKER_QEMU_CCACHE_DIR is missing $() around DOCKER_QEMU_CACHE_DIR, so
$HOME/.cache/qemu is never created.  podman then fails to mount it:

  Error: statfs /home/user/.cache/qemu: no such file or directory

It only works if the directory already exists.  Add the missing $().

Fixes: 8a1975e4d44b ("tests/docker: expose $HOME/.cache/qemu as docker volume")
Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
Message-ID: <20260915072403.347639-1-anders.roxell@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v5
  - dropped my fix in favour of Anders with a better commit (an he was
  first)
---
 tests/docker/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 74bf471f171..9237a0dbe7f 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -197,7 +197,7 @@ docker-help: docker
 # Where QEMU caches build artefacts
 DOCKER_QEMU_CACHE_DIR := $$HOME/.cache/qemu
 # Use a global constant ccache directory to speed up repetitive builds
-DOCKER_QEMU_CCACHE_DIR := DOCKER_QEMU_CACHE_DIR/docker-ccache
+DOCKER_QEMU_CCACHE_DIR := $(DOCKER_QEMU_CACHE_DIR)/docker-ccache
 
 # This rule if for directly running against an arbitrary docker target.
 # It is called by the expanded docker targets (e.g. make
-- 
2.47.3

[PATCH v5 14/17] gitlab: update the @user id mapping file

[Alex Bennée]

I needed to fix a POSIX-ism in the script as well.

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Amit Machhiwal <amachhiw@linux.ibm.com>
Message-ID: <20260914184534.706637-1-alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Amit Machhiwal <amachhiw@linux.ibm.com>
---
 .gitlab-map-auto          | 8 +++++---
 scripts/gitlab-map-update | 2 +-
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/.gitlab-map-auto b/.gitlab-map-auto
index 28ce448f1bf..3f717a15d4e 100644
--- a/.gitlab-map-auto
+++ b/.gitlab-map-auto
@@ -12,11 +12,12 @@ a1xndr	Alexander Bulekov
 adi-g15-ibm	Aditya Gupta
 agraf	Alexander Graf
 alex.williamson	Alex Williamson
-aliang1	Aihua Liang
 alistair23	Alistair Francis
+amachhiw	Amit Machhiwal
 anisinha	Ani Sinha
 anthony-linaro	Anthony Roberts
 anthonyper	Anthony PERARD
+aroxell	Anders Roxell
 berrange	Daniel P. Berrangé
 birkelund	Klaus Jensen
 bonzini	Paolo Bonzini
@@ -36,10 +37,10 @@ dwmw2	David Woodhouse
 eauger1	Eric Auger
 ebblake	Eric Blake
 edgar.iglesias	Edgar E. Iglesias
+efarman	Eric Farman
 ehabkost	Eduardo Habkost
 eldondev	Eldon
 epilys	Manos Pitsidianakis
-famzheng	Fam Zheng
 farosas	Fabiano Rosas
 gautammenghani	Gautam Menghani
 gkurz	Greg Kurz
@@ -51,7 +52,6 @@ imammedo	Igor Mammedov
 jasowang	Jason Wang
 jmacarthur	Jim MacArthur
 jsnow	John Snow
-juan.quintela	Juan Quintela
 kbastian-qemu	Bastian Koppelmann
 kmwolf	Kevin Wolf
 kostyanf14	Kostiantyn Kostiuk
@@ -67,6 +67,7 @@ mauromatteo.cascella	Mauro Matteo Cascella
 mcayland	Mark Cave-Ayland
 mdroth	Michael Roth
 mediouni-m	M. Mediouni
+mjrosato	Matthew Rosato
 mjt0k	Michael Tokarev
 mstredhat	MST
 npiggin	npiggin
@@ -79,6 +80,7 @@ pipo.sk	Peter Krempa
 pkrempa	Peter Krempa (work)
 pm215	Peter Maydell
 qemu-janitor	Qemu Janitor
+raphael.s.norwitz	Raphael Norwitz
 rathc	Chinmay Rath
 rth7680	Richard Henderson
 schoenebeck	Christian Schoenebeck
diff --git a/scripts/gitlab-map-update b/scripts/gitlab-map-update
index c134c226aa2..743ad11900d 100755
--- a/scripts/gitlab-map-update
+++ b/scripts/gitlab-map-update
@@ -33,7 +33,7 @@ glab api --paginate /projects/qemu-project%2fqemu/members | \
     .gitlab-map-auto.tmp
 test $? != 0 && fail=1
 
-if test $fail == 0
+if test $fail -eq 0
 then
   grep '^#' .gitlab-map-auto > .gitlab-map-auto.new
   LC_ALL=C sort .gitlab-map-auto.tmp | uniq >> .gitlab-map-auto.new
-- 
2.47.3

[PATCH v5 15/17] tests/tcg: add console property to docker image custom target

[Alex Bennée]

According to the meson docs this serialises the target in a special
"console" pool which also disables the buffering of stdout. This helps
avoid the appearance of a massive stall when a docker image is built.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/tcg/meson.build | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/tcg/meson.build b/tests/tcg/meson.build
index 053ef147df0..95f533b0ab5 100644
--- a/tests/tcg/meson.build
+++ b/tests/tcg/meson.build
@@ -265,6 +265,7 @@ foreach target, plan: tcg_tests
       endif
       t = custom_target(image_name, command: cmd,
                         build_by_default: false,
+                        console: true,
                         output: 'no_output_' + image_name)
       image_targets += {cc_dockerfile: t}
     endif
-- 
2.47.3

[PATCH v5 16/17] tests/tcg: Fix building outside of the source tree

[Alex Bennée]

From: Ilya Leoshkevich <iii@linux.ibm.com>

Running TCG tests from a build directory that is outside of the source
tree fails with:

    [...]/qemu/tests/tcg/multiarch/libs/float_helpers.c:228:1: fatal error: opening dependency file tests/tcg/s390x-linux-user-float_convs.d: No such file or directory

This is because the build directory is not mounted into the container.
Add this mount. Do not bother with preventing overlapping mounts, they
are harmless.

Fixes: 27939ad414ca ("tests/tcg/meson.build: introduce cc_dockerfile and cc_docker_host_arch")
Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-ID: <20260915110855.491985-1-iii@linux.ibm.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/tcg/meson.build | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/tcg/meson.build b/tests/tcg/meson.build
index 95f533b0ab5..a5e0462732d 100644
--- a/tests/tcg/meson.build
+++ b/tests/tcg/meson.build
@@ -277,7 +277,7 @@ foreach target, plan: tcg_tests
       mount = mount + ':' + mount
       here = meson.project_build_root()
       cc = [docker_wrapper, 'run', '--run-as-current-user', '--',
-            '-w', here, '-v', mount,
+            '-w', here, '-v', here + ':' + here, '-v', mount,
             'qemu/' + cc_dockerfile, plan['cc']]
       has_cc = true
     endif
-- 
2.47.3

[PATCH v5 17/17] gitlab: drop check-tcg from clang-system job

[Alex Bennée]

We have no cross-compilers in the fedora image so don't bother trying
to run the tests.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .gitlab-ci.d/buildtest.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 5f21703645b..a97cc0b20b8 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -441,7 +441,7 @@ clang-system:
     CONFIGURE_ARGS: --cc=clang --cxx=clang++ --enable-ubsan
       --extra-cflags=-fno-sanitize-recover=undefined
     TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu s390x-softmmu
-    MAKE_CHECK_ARGS: check-qtest check-tcg
+    MAKE_CHECK_ARGS: check-qtest
 
 clang-user:
   extends: .native_build_job_template
-- 
2.47.3

Re: [PATCH v5 08/17] tests/tcg: don't define tests when missing compilers

[Pierrick Bouvier]

On 9/17/2026 9:55 AM, Alex Bennée wrote:
> Skip defining the tests if we don't have a compiler to build them.
> Otherwise we add a bunch of noise to the reported results. We still do
> report when we skip because the compiler doesn't have the features we
> need to build.
> 
> Suggested-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> ---
> v5
>   - alternate patch from Pierrick which keeps validation
> ---
>  tests/tcg/meson.build | 12 ++++++++----
>  1 file changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/tests/tcg/meson.build b/tests/tcg/meson.build
> index 814f2f8cd8c..5819fdf0bf3 100644
> --- a/tests/tcg/meson.build
> +++ b/tests/tcg/meson.build
> @@ -499,10 +499,14 @@ foreach target, plan: tcg_tests
>          cmd = ['sh', '-c', 'echo "' + skip_test + '" && exit 77']
>        endif
>  
> -      test(test_name, env, args: cmd,
> -           depends: test_depends,
> -           timeout: 120,
> -           suite: ['tcg', 'tcg-' + target])
> +      # Do not add a skipped test when compiler is missing,
> +      # though we still check all tests are correctly defined.
> +      if has_cc
> +        test(test_name, env, args: cmd,
> +             depends: test_depends,
> +             timeout: 120,
> +             suite: ['tcg', 'tcg-' + target])
> +      endif
>        added_tests += {test_name: true}
>      endforeach
>    endforeach

To help series get pulled as it's ready,
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

Re: [PATCH v5 17/17] gitlab: drop check-tcg from clang-system job

[Pierrick Bouvier]

On 9/17/2026 9:56 AM, Alex Bennée wrote:
> We have no cross-compilers in the fedora image so don't bother trying
> to run the tests.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  .gitlab-ci.d/buildtest.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 5f21703645b..a97cc0b20b8 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -441,7 +441,7 @@ clang-system:
>      CONFIGURE_ARGS: --cc=clang --cxx=clang++ --enable-ubsan
>        --extra-cflags=-fno-sanitize-recover=undefined
>      TARGETS: alpha-softmmu arm-softmmu m68k-softmmu mips64-softmmu s390x-softmmu
> -    MAKE_CHECK_ARGS: check-qtest check-tcg
> +    MAKE_CHECK_ARGS: check-qtest
>  
>  clang-user:
>    extends: .native_build_job_template

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

In the future, we could use clang with -target X, however most of tcg
tests don't build with this compiler.

Re: [PATCH v5 00/17] testing: various docker clean-ups post meson tcg conversion (pre-PR)

[Pierrick Bouvier]

On 9/17/2026 9:55 AM, Alex Bennée wrote:
> This cleans up a few bits and pieces although doesn't totally solve
> the issue of an apparent stall while meson builds the docker image.
> However with these changes there should be no difference between the
> meson build and the equivalent "make docker-image-foo" invocation.
> 
> We also finally get rid of the ill-advised magic caching support in
> docker.py.
> 
> v5
>  - fixup a bunch of rebase damage
>  - swap out my ccache fix for Anders
>  - rename revert to "configure: ensure it also honours container configuration"
>  - use Pierricks solution for avoid too many SKIPs while validating config
>

It seems that this version removes patch:
Revert "tests/tcg/meson.build: implement ./configure -cross-cc-* options"

That's fine and correct, but we're missing two things:
- cross-cc-cflags and prefix should be passed to config.mak also
(similar to the merge for container options). Not strictly needed as I'm
not sure anyone needs this "extra" flexibility for roms/vdso, but needed
if we want to make sure we don't break anything for anyone.
- scripts/meson_buildoptions.sh update that was missing originally for
those meson options.

For the rest, it looks good and ready!

Regards,
Pierrick

Re: [PATCH v5 13/17] tests/docker: fix ccache directory variable expansion

[Pierrick Bouvier]

On 9/17/2026 9:55 AM, Alex Bennée wrote:
> From: Anders Roxell <anders.roxell@linaro.org>
> 
> DOCKER_QEMU_CCACHE_DIR is missing $() around DOCKER_QEMU_CACHE_DIR, so
> $HOME/.cache/qemu is never created.  podman then fails to mount it:
> 
>   Error: statfs /home/user/.cache/qemu: no such file or directory
> 
> It only works if the directory already exists.  Add the missing $().
> 
> Fixes: 8a1975e4d44b ("tests/docker: expose $HOME/.cache/qemu as docker volume")
> Signed-off-by: Anders Roxell <anders.roxell@linaro.org>
> Message-ID: <20260915072403.347639-1-anders.roxell@linaro.org>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> ---
> v5
>   - dropped my fix in favour of Anders with a better commit (an he was
>   first)
> ---
>  tests/docker/Makefile.include | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

Re: [PATCH v5 00/17] testing: various docker clean-ups post meson tcg conversion (pre-PR)

[Alex Bennée]

Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> On 9/17/2026 9:55 AM, Alex Bennée wrote:
>> This cleans up a few bits and pieces although doesn't totally solve
>> the issue of an apparent stall while meson builds the docker image.
>> However with these changes there should be no difference between the
>> meson build and the equivalent "make docker-image-foo" invocation.
>> 
>> We also finally get rid of the ill-advised magic caching support in
>> docker.py.
>> 
>> v5
>>  - fixup a bunch of rebase damage
>>  - swap out my ccache fix for Anders
>>  - rename revert to "configure: ensure it also honours container configuration"
>>  - use Pierricks solution for avoid too many SKIPs while validating config
>>
>
> It seems that this version removes patch:
> Revert "tests/tcg/meson.build: implement ./configure -cross-cc-* options"
>
> That's fine and correct, but we're missing two things:
> - cross-cc-cflags and prefix should be passed to config.mak also
> (similar to the merge for container options). Not strictly needed as I'm
> not sure anyone needs this "extra" flexibility for roms/vdso, but needed
> if we want to make sure we don't break anything for anyone.

The rom builds are outside the main makefile anyway so they just include
random bits as they want.

> - scripts/meson_buildoptions.sh update that was missing originally for
> those meson options.

We need to do something about it because at various times
"update-buildoptions" does and doesn't change things. Sometimes there
are more options than I've been messing with. It is super confusing.

>
> For the rest, it looks good and ready!
>
> Regards,
> Pierrick

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v5 00/17] testing: various docker clean-ups post meson tcg conversion (pre-PR)

[Pierrick Bouvier]

On 9/17/2026 10:45 AM, Alex Bennée wrote:
> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
> 
>> On 9/17/2026 9:55 AM, Alex Bennée wrote:
>>> This cleans up a few bits and pieces although doesn't totally solve
>>> the issue of an apparent stall while meson builds the docker image.
>>> However with these changes there should be no difference between the
>>> meson build and the equivalent "make docker-image-foo" invocation.
>>>
>>> We also finally get rid of the ill-advised magic caching support in
>>> docker.py.
>>>
>>> v5
>>>  - fixup a bunch of rebase damage
>>>  - swap out my ccache fix for Anders
>>>  - rename revert to "configure: ensure it also honours container configuration"
>>>  - use Pierricks solution for avoid too many SKIPs while validating config
>>>
>>
>> It seems that this version removes patch:
>> Revert "tests/tcg/meson.build: implement ./configure -cross-cc-* options"
>>
>> That's fine and correct, but we're missing two things:
>> - cross-cc-cflags and prefix should be passed to config.mak also
>> (similar to the merge for container options). Not strictly needed as I'm
>> not sure anyone needs this "extra" flexibility for roms/vdso, but needed
>> if we want to make sure we don't break anything for anyone.
> 
> The rom builds are outside the main makefile anyway so they just include
> random bits as they want.
> 
>> - scripts/meson_buildoptions.sh update that was missing originally for
>> those meson options.
> 
> We need to do something about it because at various times
> "update-buildoptions" does and doesn't change things. Sometimes there
> are more options than I've been messing with. It is super confusing.
>

Agree, in my personal experience, I never could regenerate it through
Makefile dependencies, even if it's supposed to work.

What we could do is generate it directly from meson_options.txt, instead
of meson introspect. It should be quite easy as meson_options.txt can be
"included" as a python file, and we can provide option() function to
aggregate them.

This way, our only dependency becomes python, which is already a dep
anyway, and not our full venv + meson. We could directly generate this
in configure, without any need to keep the file checked in our tree.
This way, it would always be up to date by design.

I'll let Paolo judge if that's a good idea or if I missed something
important.

>>
>> For the rest, it looks good and ready!
>>
>> Regards,
>> Pierrick
> 

Regards,
Pierrick

Re: [PATCH v5 00/17] testing: various docker clean-ups post meson tcg conversion (pre-PR)

[Alex Bennée]

Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> On 9/17/2026 9:55 AM, Alex Bennée wrote:
>> This cleans up a few bits and pieces although doesn't totally solve
>> the issue of an apparent stall while meson builds the docker image.
>> However with these changes there should be no difference between the
>> meson build and the equivalent "make docker-image-foo" invocation.
>> 
>> We also finally get rid of the ill-advised magic caching support in
>> docker.py.
>> 
>> v5
>>  - fixup a bunch of rebase damage
>>  - swap out my ccache fix for Anders
>>  - rename revert to "configure: ensure it also honours container configuration"
>>  - use Pierricks solution for avoid too many SKIPs while validating config
>>
>
> It seems that this version removes patch:
> Revert "tests/tcg/meson.build: implement ./configure -cross-cc-* options"
>
> That's fine and correct, but we're missing two things:
> - cross-cc-cflags and prefix should be passed to config.mak also
> (similar to the merge for container options). Not strictly needed as I'm
> not sure anyone needs this "extra" flexibility for roms/vdso, but needed
> if we want to make sure we don't break anything for anyone.
> - scripts/meson_buildoptions.sh update that was missing originally for
> those meson options.

See: 20260917185322.3457036-1-alex.bennee@linaro.org

>
> For the rest, it looks good and ready!
>
> Regards,
> Pierrick

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Pierrick Bouvier]

On 9/17/2026 9:55 AM, Alex Bennée wrote:
> The caching logic was a remnant of an older time when we had multiple
> shared layers building that went into the final image. It was arguable
> that it worked back then but it is definitely not needed now so into
> the bin it goes.
> 
> Tested-by: Richard Henderson <richard.henderson@linaro.org>
> Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  tests/docker/docker.py | 148 ++++++++++++++---------------------------
>  1 file changed, 49 insertions(+), 99 deletions(-)
>

After testing this series, it's not a good idea to include this patch.

It forces a systematic rebuild, even when image is already. This has
three consequences:
- verbosity: we see build log systematically
- slower: running a noop rebuild takes time
- network dependency: docker.py --registry triggers a pull, which fails
if there is no network. It breaks offline make check.
Also, it slows down things too.

So, let's drop this patch and keep the caching mechanism for now, it's
still useful/needed.

Regards,
Pierrick

Re: [PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Alex Bennée]

Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> On 9/17/2026 9:55 AM, Alex Bennée wrote:
>> The caching logic was a remnant of an older time when we had multiple
>> shared layers building that went into the final image. It was arguable
>> that it worked back then but it is definitely not needed now so into
>> the bin it goes.
>> 
>> Tested-by: Richard Henderson <richard.henderson@linaro.org>
>> Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>>  tests/docker/docker.py | 148 ++++++++++++++---------------------------
>>  1 file changed, 49 insertions(+), 99 deletions(-)
>>
>
> After testing this series, it's not a good idea to include this patch.
>
> It forces a systematic rebuild, even when image is already. This has
> three consequences:
> - verbosity: we see build log systematically

You asked me to drop the verbose/quiet switch patch which addresses this.

> - slower: running a noop rebuild takes time

how much time? it flies past when I run a rebuild.

> - network dependency: docker.py --registry triggers a pull, which fails
> if there is no network. It breaks offline make check.
> Also, it slows down things too.

we should be able to gate the registry on network access.

>
> So, let's drop this patch and keep the caching mechanism for now, it's
> still useful/needed.
>
> Regards,
> Pierrick

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Pierrick Bouvier]

On 9/17/2026 2:49 PM, Alex Bennée wrote:
> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
> 
>> On 9/17/2026 9:55 AM, Alex Bennée wrote:
>>> The caching logic was a remnant of an older time when we had multiple
>>> shared layers building that went into the final image. It was arguable
>>> that it worked back then but it is definitely not needed now so into
>>> the bin it goes.
>>>
>>> Tested-by: Richard Henderson <richard.henderson@linaro.org>
>>> Reviewed-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> ---
>>>  tests/docker/docker.py | 148 ++++++++++++++---------------------------
>>>  1 file changed, 49 insertions(+), 99 deletions(-)
>>>
>>
>> After testing this series, it's not a good idea to include this patch.
>>
>> It forces a systematic rebuild, even when image is already. This has
>> three consequences:
>> - verbosity: we see build log systematically
> 
> You asked me to drop the verbose/quiet switch patch which addresses this.
>

Which is good, because it was hiding the problem.
I was waiting to see the series is in good shape before testing it.

>> - slower: running a noop rebuild takes time
> 
> how much time? it flies past when I run a rebuild.
>

For a no-op tcg-tests build (i.e. with tests already built):

- With current patch:
time ninja -C tcg-tests
...
verbose pull and build
320 lines of output
...
real    0m15.817s
user    0m1.342s
sys     0m0.840s

Without the current patch (current behavior):
time ninja -C tcg-tests
[0/534] Generating tests/tcg/image-debian-all-test-cross with a custom
command
[1/288] Generating tests/tcg/image-debian-hexagon-cross with a custom
command
[2/193] Generating tests/tcg/image-debian-loongarch-cross with a custom
command
[3/156] Generating tests/tcg/image-debian-tricore-cross with a custom
command
[4/135] Generating tests/tcg/image-debian-amd64-cross with a custom command
[5/84] Generating tests/tcg/image-debian-xtensa-cross with a custom command
real    0m0.704s
user    0m0.504s
sys     0m0.132s

>> - network dependency: docker.py --registry triggers a pull, which fails
>> if there is no network. It breaks offline make check.
>> Also, it slows down things too.
> 
> we should be able to gate the registry on network access.
>

It's not as simple as "is_network_connected", there might be timeout
involved in case network has issues, amplifying the problem.

>>
>> So, let's drop this patch and keep the caching mechanism for now, it's
>> still useful/needed.
>>
>> Regards,
>> Pierrick
> 

Re: [PATCH v5 00/17] testing: various docker clean-ups post meson tcg conversion (pre-PR)

[Pierrick Bouvier]

On 9/17/2026 9:55 AM, Alex Bennée wrote:
> This cleans up a few bits and pieces although doesn't totally solve
> the issue of an apparent stall while meson builds the docker image.
> However with these changes there should be no difference between the
> meson build and the equivalent "make docker-image-foo" invocation.
> 
> We also finally get rid of the ill-advised magic caching support in
> docker.py.
> 

In addition to current series, please find here additional fixes,
notably for John's issue on Fedora:
https://lore.kernel.org/qemu-devel/20260917221117.274619-2-pierrick.bouvier@oss.qualcomm.com/T/#

It can be directly applied on top of current series.

Regards,
Pierrick

Re: [PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Richard Henderson]

On 9/17/26 12:00, Pierrick Bouvier wrote:
> On 9/17/2026 2:49 PM, Alex Bennée wrote:
>> how much time? it flies past when I run a rebuild.
>>
> 
> For a no-op tcg-tests build (i.e. with tests already built):
> 
> - With current patch:
> time ninja -C tcg-tests
> ...
> verbose pull and build
> 320 lines of output
> ...
> real    0m15.817s
> user    0m1.342s
> sys     0m0.840s

The variance from the middle of the pacific ocean, and associated 
network latency, is between 20 and 27 seconds.

Amusingly, with the network actively disconnected (airplane mode), the 
no-op build completes in 1 to 2 seconds.


r~

Re: [PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Pierrick Bouvier]

On 9/17/2026 3:18 PM, Richard Henderson wrote:
> On 9/17/26 12:00, Pierrick Bouvier wrote:
>> On 9/17/2026 2:49 PM, Alex Bennée wrote:
>>> how much time? it flies past when I run a rebuild.
>>>
>>
>> For a no-op tcg-tests build (i.e. with tests already built):
>>
>> - With current patch:
>> time ninja -C tcg-tests
>> ...
>> verbose pull and build
>> 320 lines of output
>> ...
>> real    0m15.817s
>> user    0m1.342s
>> sys     0m0.840s
> 
> The variance from the middle of the pacific ocean, and associated
> network latency, is between 20 and 27 seconds.
> 
> Amusingly, with the network actively disconnected (airplane mode), the
> no-op build completes in 1 to 2 seconds.
>

Funny, on my side if hangs a long time (timeout 30s), then retry 3 times
per image. I tried it on WSL on my windows laptop.

> 
> r~

Re: [PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Alex Bennée]

Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> On 9/17/2026 3:18 PM, Richard Henderson wrote:
>> On 9/17/26 12:00, Pierrick Bouvier wrote:
>>> On 9/17/2026 2:49 PM, Alex Bennée wrote:
>>>> how much time? it flies past when I run a rebuild.
>>>>
>>>
>>> For a no-op tcg-tests build (i.e. with tests already built):
>>>
>>> - With current patch:
>>> time ninja -C tcg-tests
>>> ...
>>> verbose pull and build
>>> 320 lines of output
>>> ...
>>> real    0m15.817s
>>> user    0m1.342s
>>> sys     0m0.840s

  ✗  time ninja
  [1/114] Generating qemu-version.h with a custom command (wrapped by meson to capture output)

  ________________________________________________________
  Executed in  515.16 millis    fish           external
     usr time  433.82 millis  968.00 micros  432.85 millis
     sys time   86.05 millis  253.00 micros   85.80 millis

Not sure whats going on there then. The whole tcg suite completes in
under a minute (and would be shorter if i386-linux-user-test-i386-fprem
and riscv64-softmmu-plugin-discons-with-interruptedmemory where in the
thorough test suite).

>> 
>> The variance from the middle of the pacific ocean, and associated
>> network latency, is between 20 and 27 seconds.
>> 
>> Amusingly, with the network actively disconnected (airplane mode), the
>> no-op build completes in 1 to 2 seconds.
>>
>
> Funny, on my side if hangs a long time (timeout 30s), then retry 3 times
> per image. I tried it on WSL on my windows laptop.
>
>> 
>> r~

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Pierrick Bouvier]

On 9/18/2026 12:30 AM, Alex Bennée wrote:
> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
> 
>> On 9/17/2026 3:18 PM, Richard Henderson wrote:
>>> On 9/17/26 12:00, Pierrick Bouvier wrote:
>>>> On 9/17/2026 2:49 PM, Alex Bennée wrote:
>>>>> how much time? it flies past when I run a rebuild.
>>>>>
>>>>
>>>> For a no-op tcg-tests build (i.e. with tests already built):
>>>>
>>>> - With current patch:
>>>> time ninja -C tcg-tests
>>>> ...
>>>> verbose pull and build
>>>> 320 lines of output
>>>> ...
>>>> real    0m15.817s
>>>> user    0m1.342s
>>>> sys     0m0.840s
> 
>   ✗  time ninja
>   [1/114] Generating qemu-version.h with a custom command (wrapped by meson to capture output)
> 
>   ________________________________________________________
>   Executed in  515.16 millis    fish           external
>      usr time  433.82 millis  968.00 micros  432.85 millis
>      sys time   86.05 millis  253.00 micros   85.80 millis
> 
> Not sure whats going on there then. The whole tcg suite completes in
> under a minute (and would be shorter if i386-linux-user-test-i386-fprem
> and riscv64-softmmu-plugin-discons-with-interruptedmemory where in the
> thorough test suite).
>

You need to run "ninja tcg-tests" explicitly, as tests (and images) are
not built by default.

>>>
>>> The variance from the middle of the pacific ocean, and associated
>>> network latency, is between 20 and 27 seconds.
>>>
>>> Amusingly, with the network actively disconnected (airplane mode), the
>>> no-op build completes in 1 to 2 seconds.
>>>
>>
>> Funny, on my side if hangs a long time (timeout 30s), then retry 3 times
>> per image. I tried it on WSL on my windows laptop.
>>
>>>
>>> r~
> 

Re: [PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Pierrick Bouvier]

On 9/18/2026 1:05 AM, Pierrick Bouvier wrote:
> On 9/18/2026 12:30 AM, Alex Bennée wrote:
>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>>
>>> On 9/17/2026 3:18 PM, Richard Henderson wrote:
>>>> On 9/17/26 12:00, Pierrick Bouvier wrote:
>>>>> On 9/17/2026 2:49 PM, Alex Bennée wrote:
>>>>>> how much time? it flies past when I run a rebuild.
>>>>>>
>>>>>
>>>>> For a no-op tcg-tests build (i.e. with tests already built):
>>>>>
>>>>> - With current patch:
>>>>> time ninja -C tcg-tests
>>>>> ...
>>>>> verbose pull and build
>>>>> 320 lines of output
>>>>> ...
>>>>> real    0m15.817s
>>>>> user    0m1.342s
>>>>> sys     0m0.840s
>>
>>   ✗  time ninja
>>   [1/114] Generating qemu-version.h with a custom command (wrapped by meson to capture output)
>>
>>   ________________________________________________________
>>   Executed in  515.16 millis    fish           external
>>      usr time  433.82 millis  968.00 micros  432.85 millis
>>      sys time   86.05 millis  253.00 micros   85.80 millis
>>
>> Not sure whats going on there then. The whole tcg suite completes in
>> under a minute (and would be shorter if i386-linux-user-test-i386-fprem
>> and riscv64-softmmu-plugin-discons-with-interruptedmemory where in the
>> thorough test suite).
>>
> 
> You need to run "ninja tcg-tests" explicitly, as tests (and images) are
> not built by default.
>

To give a final insight before bed time and the PR, the current "magic"
caching implemented in docker.py is simply computing a hash of
dockerfile content, and if a matching image exists (i.e. with a label
having this hash), it skips the build. Please note that ninja will
always call docker.py build, and the trick is that docker.py selectively
skips it.

That's a common and safe optimization for containers which don't have
external context (varying build-arg or imported files with ADD/COPY).

The current series is ready (ideally, with additional
20260917221117.274619-1-pierrick.bouvier@oss.qualcomm.com) and PR can be
sent to share the fixes with everyone. It solves all problems that have
been reported.

Let's drop the current patch, and you can take a look at it later to
convince yourself it should not be removed.

Regards,
Pierrick

Re: [PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Alex Bennée]

Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:

> On 9/18/2026 12:30 AM, Alex Bennée wrote:
>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>> 
>>> On 9/17/2026 3:18 PM, Richard Henderson wrote:
>>>> On 9/17/26 12:00, Pierrick Bouvier wrote:
>>>>> On 9/17/2026 2:49 PM, Alex Bennée wrote:
>>>>>> how much time? it flies past when I run a rebuild.
>>>>>>
>>>>>
>>>>> For a no-op tcg-tests build (i.e. with tests already built):
>>>>>
>>>>> - With current patch:
>>>>> time ninja -C tcg-tests
>>>>> ...
>>>>> verbose pull and build
>>>>> 320 lines of output
>>>>> ...
>>>>> real    0m15.817s
>>>>> user    0m1.342s
>>>>> sys     0m0.840s
>> 
>>   ✗  time ninja
>>   [1/114] Generating qemu-version.h with a custom command (wrapped by meson to capture output)
>> 
>>   ________________________________________________________
>>   Executed in  515.16 millis    fish           external
>>      usr time  433.82 millis  968.00 micros  432.85 millis
>>      sys time   86.05 millis  253.00 micros   85.80 millis
>> 
>> Not sure whats going on there then. The whole tcg suite completes in
>> under a minute (and would be shorter if i386-linux-user-test-i386-fprem
>> and riscv64-softmmu-plugin-discons-with-interruptedmemory where in the
>> thorough test suite).
>>
>
> You need to run "ninja tcg-tests" explicitly, as tests (and images) are
> not built by default.

What is that target trying to do? Because when I run:

  ./run meson test --suite tcg

it still ends up rebuilding the test binaries. 

>
>>>>
>>>> The variance from the middle of the pacific ocean, and associated
>>>> network latency, is between 20 and 27 seconds.
>>>>
>>>> Amusingly, with the network actively disconnected (airplane mode), the
>>>> no-op build completes in 1 to 2 seconds.
>>>>
>>>
>>> Funny, on my side if hangs a long time (timeout 30s), then retry 3 times
>>> per image. I tried it on WSL on my windows laptop.
>>>
>>>>
>>>> r~
>> 

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Pierrick Bouvier]

On 9/18/2026 4:57 AM, Alex Bennée wrote:
> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
> 
>> On 9/18/2026 12:30 AM, Alex Bennée wrote:
>>> Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com> writes:
>>>
>>>> On 9/17/2026 3:18 PM, Richard Henderson wrote:
>>>>> On 9/17/26 12:00, Pierrick Bouvier wrote:
>>>>>> On 9/17/2026 2:49 PM, Alex Bennée wrote:
>>>>>>> how much time? it flies past when I run a rebuild.
>>>>>>>
>>>>>>
>>>>>> For a no-op tcg-tests build (i.e. with tests already built):
>>>>>>
>>>>>> - With current patch:
>>>>>> time ninja -C tcg-tests
>>>>>> ...
>>>>>> verbose pull and build
>>>>>> 320 lines of output
>>>>>> ...
>>>>>> real    0m15.817s
>>>>>> user    0m1.342s
>>>>>> sys     0m0.840s
>>>
>>>   ✗  time ninja
>>>   [1/114] Generating qemu-version.h with a custom command (wrapped by meson to capture output)
>>>
>>>   ________________________________________________________
>>>   Executed in  515.16 millis    fish           external
>>>      usr time  433.82 millis  968.00 micros  432.85 millis
>>>      sys time   86.05 millis  253.00 micros   85.80 millis
>>>
>>> Not sure whats going on there then. The whole tcg suite completes in
>>> under a minute (and would be shorter if i386-linux-user-test-i386-fprem
>>> and riscv64-softmmu-plugin-discons-with-interruptedmemory where in the
>>> thorough test suite).
>>>
>>
>> You need to run "ninja tcg-tests" explicitly, as tests (and images) are
>> not built by default.
> 
> What is that target trying to do? Because when I run:
>

tcg-tests will build only tests (and their dependencies, container images).

tcg/tests/meson.build
```
if exe_name not in built_tests
    exe = custom_target(exe_name,
                        input: file,
                        command: [cc, '@INPUT@',
                                  '-o', '@OUTPUT@', '-static',
                                  '-MMD', '-MF', '@DEPFILE@',
                                  '-Wall', '-Werror',
                                  '-O0', '-g', '-fno-strict-aliasing',
                                  cflags],
                        depfile: exe_name + '.d',
                        output: exe_name + '.test',
                        depends: exe_depends,
                        depend_files: build_test_depend_files,
                        build_by_default: false)
    exe_targets += exe
endif

...

if exe_targets.length() > 0
  # finally create a top level target to rebuild all tests
  alias_target('tcg-tests', exe_targets)
endif
```

build_test_depend_files contains container images, and eventually some
generated files given in "depends" field of test declaration. It's used
for x86 tests who need to generate headers.

Test compilation generates depfile (.d), which meson reintegrates
automatically, so test depends on all their headers correctly.

```
build_test_depends = []
build_test_depends_files = []
...
      build_test_depends += image_targets[cc_dockerfile]
      build_test_depend_files += dockerfile
...
exe_depends = build_test_depends
if 'depends' in setup
    exe_depends += setup['depends']
endif

exe..
   depends: exe_depends,
   depend_files: build_test_depend_files,
# meson has distinct fields for target dependencies and file
# dependencies, which is a bit confusing IMHO.
```

>   ./run meson test --suite tcg
>

This target will build test dependencies, and run them also.

```
test(test_name, env, args: cmd,
     depends: test_depends,
     timeout: 120,
     suite: ['tcg', 'tcg-' + target])
```

test_depends contains the test binary itself (as seen above), but also
qemu binaries, plugins.

```
test_depends = []
...
if 'plugin_test' in setup
   test_depends += plugin
endif
...
test_depends += qemu
```

Also, meson adds an implicit dependency on every target included in cmd
(args: cmd) array, which includes transitively test binaries and all
their deps.

> it still ends up rebuilding the test binaries. 
>

So from all code above, it should now be clear that:
   `meson --suite tcg`
will automatically build all that is needed (and no more - like no other
test binaries). Also, if you modify an existing dockerfile, it will
automatically rebuilds all tests associated for you.

Same goes for any subsuite: tcg-aarch64-softmmu, which only build
qemu-system-aarch64, needed container images, and only test binaries.

Beyond a better maintainability, one of the goal of original series was
to have *complete* dependencies, so we never need to add fake "all"
dependencies on Makefile level, which is why I insisted to remove the
patch that was doing it for images in this series.

Hope it's more clear now!

>>
>>>>>
>>>>> The variance from the middle of the pacific ocean, and associated
>>>>> network latency, is between 20 and 27 seconds.
>>>>>
>>>>> Amusingly, with the network actively disconnected (airplane mode), the
>>>>> no-op build completes in 1 to 2 seconds.
>>>>>
>>>>
>>>> Funny, on my side if hangs a long time (timeout 30s), then retry 3 times
>>>> per image. I tried it on WSL on my windows laptop.
>>>>
>>>>>
>>>>> r~
>>>
> 

Re: [PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Richard Henderson]

On 9/18/26 07:07, Pierrick Bouvier wrote:
>    # finally create a top level target to rebuild all tests
>    alias_target('tcg-tests', exe_targets)

FWIW, that used to be called build-tcg.


r~

Re: [PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Pierrick Bouvier]

On 9/18/2026 10:58 AM, Richard Henderson wrote:
> On 9/18/26 07:07, Pierrick Bouvier wrote:
>>    # finally create a top level target to rebuild all tests
>>    alias_target('tcg-tests', exe_targets)
> 
> FWIW, that used to be called build-tcg.
>

Would you prefer to change name to that?

I don't mind personally, and if some people were used to it, it makes
sense to provide the same.

> 
> r~

Re: [PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Richard Henderson]

On 9/18/26 09:22, Pierrick Bouvier wrote:
> On 9/18/2026 10:58 AM, Richard Henderson wrote:
>> On 9/18/26 07:07, Pierrick Bouvier wrote:
>>>     # finally create a top level target to rebuild all tests
>>>     alias_target('tcg-tests', exe_targets)
>>
>> FWIW, that used to be called build-tcg.
>>
> 
> Would you prefer to change name to that?
> 
> I don't mind personally, and if some people were used to it, it makes
> sense to provide the same.
If we're going to provide a fake target at all, we might as well keep it 
the same.

Though being able to run individual tests from the make/meson 
command-line is probably the biggest new feature, plausibly obviating 
most uses of "just build all the tests so I can run the one I want 
manually".


r~

Re: [PATCH v5 11/17] tests/docker: strip remaining caching logic from docker.py

[Pierrick Bouvier]

On 9/18/2026 12:53 PM, Richard Henderson wrote:
> On 9/18/26 09:22, Pierrick Bouvier wrote:
>> On 9/18/2026 10:58 AM, Richard Henderson wrote:
>>> On 9/18/26 07:07, Pierrick Bouvier wrote:
>>>>     # finally create a top level target to rebuild all tests
>>>>     alias_target('tcg-tests', exe_targets)
>>>
>>> FWIW, that used to be called build-tcg.
>>>
>>
>> Would you prefer to change name to that?
>>
>> I don't mind personally, and if some people were used to it, it makes
>> sense to provide the same.
> If we're going to provide a fake target at all, we might as well keep it
> the same.
>

We can have both, it's free.
I'll send a patch to reintroduce this name along tcg-tests (definitely
bikeshedding territory, but I feel tcg-tests is a bit better than
build-tcg).

> Though being able to run individual tests from the make/meson command-
> line is probably the biggest new feature, plausibly obviating most uses
> of "just build all the tests so I can run the one I want manually".
>

True, I even hesitated to introduce this extra target, but for
development its pretty convenient, and people are used to have it.

> 
> r~