[PATCH 0/2] build: make tests/tcg compiler detection code more generic

[PATCH 0/2] build: make tests/tcg compiler detection code more generic

[Paolo Bonzini]

Configure is trying to fall back on cross compilers for targets that
can have bi-arch or bi-endian toolchains, but there are many corner
cases where just checking the name can go wrong.  For example, the RHEL
ppc64le compiler is bi-arch and bi-endian, but multilibs are disabled.
Therefore it cannot be used to build 32-bit hosted binaries like the
linux-user TCG tests.

Trying the cross compiler first also does not work, and an example for
this is also ppc64le.  The powerpc64-linux-gnu-gcc binary from the
cross-gcc package is theoretically multilib-friendly, but it cannot
find the CRT files on a ppc64le host, because they are not in the .../le
multilib subdirectory.

This can be fixed by testing both the native compiler and the cross
compiler, and proceeding with the first one that works.  To do this,
move the compiler usability check from the tests/tcg snippet to inside
probe_target_compiler.

While at it, restrict it to just the user-mode emulation tests; if
a compiler is not able to build nostdlib freestanding binaries the
installation is broken.  This however detects a problem with some
system emulation tests that are using inttypes.h instead of stdint.h,
and not passing -ffreestanding to the compiler.  The first patch
fixes that.

Paolo

Based-on: <20220621075147.36297-1-pbonzini@redhat.com>


 configure                              | 166 +++++++++++++++----------
 tests/tcg/Makefile.target              |   1 +
 tests/tcg/aarch64/system/pauth-3.c     |   2 +-
 tests/tcg/aarch64/system/semiconsole.c |   2 +-
 tests/tcg/aarch64/system/semiheap.c    |   2 +-
 tests/tcg/multiarch/system/memory.c    |   2 +-
 6 files changed, 105 insertions(+), 70 deletions(-)

-- 
2.36.1

[PATCH 1/2] tests/tcg: compile system emulation tests as freestanding

[Paolo Bonzini]

System emulation tests do not run in a hosted environment, since they
do not link with libc.  They should only use freestanding headers
(float.h, limits.h, stdarg.h, stddef.h, stdbool.h, stdint.h,
stdalign.h, stdnoreturn.h) and should be compiled with -ffreestanding
in order to use the compiler implementation of those headers
rather than the one in libc.

Some tests are using inttypes.h instead of stdint.h, so fix that.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 tests/tcg/Makefile.target              | 1 +
 tests/tcg/aarch64/system/pauth-3.c     | 2 +-
 tests/tcg/aarch64/system/semiconsole.c | 2 +-
 tests/tcg/aarch64/system/semiheap.c    | 2 +-
 tests/tcg/multiarch/system/memory.c    | 2 +-
 5 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/tests/tcg/Makefile.target b/tests/tcg/Makefile.target
index f427a0304e..e68830af15 100644
--- a/tests/tcg/Makefile.target
+++ b/tests/tcg/Makefile.target
@@ -111,6 +111,7 @@ else
 # For softmmu targets we include a different Makefile fragement as the
 # build options for bare programs are usually pretty different. They
 # are expected to provide their own build recipes.
+EXTRA_CFLAGS += -ffreestanding
 -include $(SRC_PATH)/tests/tcg/minilib/Makefile.target
 -include $(SRC_PATH)/tests/tcg/multiarch/system/Makefile.softmmu-target
 -include $(SRC_PATH)/tests/tcg/$(TARGET_NAME)/Makefile.softmmu-target
diff --git a/tests/tcg/aarch64/system/pauth-3.c b/tests/tcg/aarch64/system/pauth-3.c
index 42eff4d5ea..77a467277b 100644
--- a/tests/tcg/aarch64/system/pauth-3.c
+++ b/tests/tcg/aarch64/system/pauth-3.c
@@ -1,4 +1,4 @@
-#include <inttypes.h>
+#include <stdint.h>
 #include <minilib.h>
 
 int main()
diff --git a/tests/tcg/aarch64/system/semiconsole.c b/tests/tcg/aarch64/system/semiconsole.c
index bfe7c9e26b..81324c639f 100644
--- a/tests/tcg/aarch64/system/semiconsole.c
+++ b/tests/tcg/aarch64/system/semiconsole.c
@@ -6,7 +6,7 @@
  * SPDX-License-Identifier: GPL-2.0-or-later
  */
 
-#include <inttypes.h>
+#include <stdint.h>
 #include <minilib.h>
 
 #define SYS_READC 0x7
diff --git a/tests/tcg/aarch64/system/semiheap.c b/tests/tcg/aarch64/system/semiheap.c
index 4ed258476d..a254bd8982 100644
--- a/tests/tcg/aarch64/system/semiheap.c
+++ b/tests/tcg/aarch64/system/semiheap.c
@@ -6,7 +6,7 @@
  * SPDX-License-Identifier: GPL-2.0-or-later
  */
 
-#include <inttypes.h>
+#include <stdint.h>
 #include <stddef.h>
 #include <minilib.h>
 
diff --git a/tests/tcg/multiarch/system/memory.c b/tests/tcg/multiarch/system/memory.c
index 41c7f66e2e..214f7d4f54 100644
--- a/tests/tcg/multiarch/system/memory.c
+++ b/tests/tcg/multiarch/system/memory.c
@@ -12,7 +12,7 @@
  *   - sign extension when loading
  */
 
-#include <inttypes.h>
+#include <stdint.h>
 #include <stdbool.h>
 #include <minilib.h>
 
-- 
2.36.1

Re: [PATCH 1/2] tests/tcg: compile system emulation tests as freestanding

[Richard Henderson]

On 6/22/22 06:47, Paolo Bonzini wrote:
> System emulation tests do not run in a hosted environment, since they
> do not link with libc.  They should only use freestanding headers
> (float.h, limits.h, stdarg.h, stddef.h, stdbool.h, stdint.h,
> stdalign.h, stdnoreturn.h) and should be compiled with -ffreestanding
> in order to use the compiler implementation of those headers
> rather than the one in libc.
> 
> Some tests are using inttypes.h instead of stdint.h, so fix that.
> 
> Signed-off-by: Paolo Bonzini<pbonzini@redhat.com>
> ---
>   tests/tcg/Makefile.target              | 1 +
>   tests/tcg/aarch64/system/pauth-3.c     | 2 +-
>   tests/tcg/aarch64/system/semiconsole.c | 2 +-
>   tests/tcg/aarch64/system/semiheap.c    | 2 +-
>   tests/tcg/multiarch/system/memory.c    | 2 +-
>   5 files changed, 5 insertions(+), 4 deletions(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

[PATCH 2/2] build: try both native and cross compilers for linux-user tests

[Paolo Bonzini]

Configure is trying to fall back on cross compilers for targets that
can have bi-arch or bi-endian toolchains, but there are many corner
cases where just checking the name can go wrong.  For example, the RHEL
ppc64le compiler is bi-arch and bi-endian, but multilibs are disabled.
Therefore it cannot be used to build 32-bit hosted binaries like the
linux-user TCG tests.

Trying the cross compiler first also does not work, and an example for
this is also ppc64le.  The powerpc64-linux-gnu-gcc binary from the
cross-gcc package is theoretically multilib-friendly, but it cannot
find the CRT files on a ppc64le host, because they are not in the .../le
multilib subdirectory.

This can be fixed by testing both the native compiler and the cross
compiler, and proceeding with the first one that works.  To do this,
move the compiler usability check from the tests/tcg snippet to inside
probe_target_compiler and, while at it, restrict it to just the user-mode
emulation tests; if a compiler is not able to build nostdlib freestanding
binaries the installation is broken.
---
 configure | 166 ++++++++++++++++++++++++++++++++----------------------
 1 file changed, 100 insertions(+), 66 deletions(-)

diff --git a/configure b/configure
index 0fd2838e82..e719afb80b 100755
--- a/configure
+++ b/configure
@@ -1868,6 +1868,7 @@ fi
 : ${cross_cc_cflags_x86_64="-m64"}
 
 compute_target_variable() {
+  eval "$2="
   if eval test -n "\"\${cross_prefix_$1}\""; then
     if eval has "\"\${cross_prefix_$1}\$3\""; then
       eval "$2=\"\${cross_prefix_$1}\$3\""
@@ -1875,8 +1876,21 @@ compute_target_variable() {
   fi
 }
 
+# probe_target_compiler TARGET TEST-PROGRAM
+#
+# Look for a compiler for the given target, either native or cross.
+# Set variables target_* if a compiler is found, and container_cross_*
+# if a Docker-based cross-compiler image is known for the target.
+# Set got_cross_cc to yes/no depending on whether a non-container-based
+# compiler was found.
+#
+# If TEST-PROGRAM is present, use it to test the usability of the
+# compiler, and also set build_static to "y" if static linking is
+# possible.
+#
 probe_target_compiler() {
   # reset all output variables
+  got_cross_cc=no
   container_image=
   container_hosts=
   container_cross_cc=
@@ -1887,14 +1901,6 @@ probe_target_compiler() {
   container_cross_objcopy=
   container_cross_ranlib=
   container_cross_strip=
-  target_cc=
-  target_ar=
-  target_as=
-  target_ld=
-  target_nm=
-  target_objcopy=
-  target_ranlib=
-  target_strip=
 
   case $1 in
     aarch64) container_hosts="x86_64 aarch64" ;;
@@ -2041,22 +2047,8 @@ probe_target_compiler() {
     : ${container_cross_strip:=${container_cross_prefix}strip}
   done
 
-  eval "target_cflags=\${cross_cc_cflags_$1}"
-  if eval test -n "\"\${cross_cc_$1}\""; then
-    if eval has "\"\${cross_cc_$1}\""; then
-      eval "target_cc=\"\${cross_cc_$1}\""
-    fi
-  else
-    compute_target_variable $1 target_cc gcc
-  fi
-  target_ccas=$target_cc
-  compute_target_variable $1 target_ar ar
-  compute_target_variable $1 target_as as
-  compute_target_variable $1 target_ld ld
-  compute_target_variable $1 target_nm nm
-  compute_target_variable $1 target_objcopy objcopy
-  compute_target_variable $1 target_ranlib ranlib
-  compute_target_variable $1 target_strip strip
+  local t try
+  try=cross
   case "$1:$cpu" in
     aarch64_be:aarch64 | \
     armeb:arm | \
@@ -2065,27 +2057,89 @@ probe_target_compiler() {
     ppc*:ppc64 | \
     sparc:sparc64 | \
     "$cpu:$cpu")
-      : ${target_cc:=$cc}
-      : ${target_ccas:=$ccas}
-      : ${target_as:=$as}
-      : ${target_ld:=$ld}
-      : ${target_ar:=$ar}
-      : ${target_as:=$as}
-      : ${target_ld:=$ld}
-      : ${target_nm:=$nm}
-      : ${target_objcopy:=$objcopy}
-      : ${target_ranlib:=$ranlib}
-      : ${target_strip:=$strip}
-      ;;
+      try='native cross' ;;
   esac
-  if test -n "$target_cc"; then
-    case $1 in
-      i386|x86_64)
-        if $target_cc --version | grep -qi "clang"; then
-          unset target_cc
+  eval "target_cflags=\${cross_cc_cflags_$1}"
+  for t in $try; do
+    case $t in
+    native)
+      target_cc=$cc
+      target_ccas=$ccas
+      target_ar=$ar
+      target_as=$as
+      target_ld=$ld
+      target_nm=$nm
+      target_objcopy=$objcopy
+      target_ranlib=$ranlib
+      target_strip=$strip
+      ;;
+    cross)
+      target_cc=
+      if eval test -n "\"\${cross_cc_$1}\""; then
+        if eval has "\"\${cross_cc_$1}\""; then
+          eval "target_cc=\"\${cross_cc_$1}\""
         fi
-        ;;
+      else
+        compute_target_variable $1 target_cc gcc
+      fi
+      target_ccas=$target_cc
+      compute_target_variable $1 target_ar ar
+      compute_target_variable $1 target_as as
+      compute_target_variable $1 target_ld ld
+      compute_target_variable $1 target_nm nm
+      compute_target_variable $1 target_objcopy objcopy
+      compute_target_variable $1 target_ranlib ranlib
+      compute_target_variable $1 target_strip strip
+      ;;
     esac
+
+    if test -n "$target_cc"; then
+      case $1 in
+        i386|x86_64)
+          if $target_cc --version | grep -qi "clang"; then
+            continue
+          fi
+          ;;
+      esac
+    elif test -n "$target_as" && test -n "$target_ld"; then
+      # Special handling for assembler only targets
+      case $target in
+        tricore-softmmu)
+          build_static=
+          got_cross_cc=yes
+          break
+          ;;
+        *)
+          continue
+          ;;
+      esac
+    else
+      continue
+    fi
+
+    if test $# = 2 && do_compiler "$target_cc" $target_cflags -o $TMPE $TMPC -static ; then
+      build_static=y
+      got_cross_cc=yes
+      break
+    fi
+    if test $# = 1 || do_compiler "$target_cc" $target_cflags -o $TMPE $TMPC ; then
+      build_static=
+      got_cross_cc=yes
+      break
+    fi
+  done
+  if test $got_cross_cc != yes; then
+    build_static=
+    target_cc=
+    target_ccas=
+    target_cflags=
+    target_ar=
+    target_as=
+    target_ld=
+    target_nm=
+    target_objcopy=
+    target_ranlib=
+    target_strip=
   fi
 }
 
@@ -2488,9 +2542,10 @@ tcg_tests_targets=
 for target in $target_list; do
   arch=${target%%-*}
 
-  probe_target_compiler ${arch}
   config_target_mak=tests/tcg/config-$target.mak
 
+  write_c_skeleton
+
   echo "# Automatically generated by configure - do not modify" > $config_target_mak
   echo "TARGET_NAME=$arch" >> $config_target_mak
   case $target in
@@ -2501,35 +2556,14 @@ for target in $target_list; do
     *-softmmu)
       test -f $source_path/tests/tcg/$arch/Makefile.softmmu-target || continue
       qemu="qemu-system-$arch"
+      probe_target_compiler ${arch}
       ;;
     *-linux-user|*-bsd-user)
       qemu="qemu-$arch"
+      probe_target_compiler ${arch} $TMPC
       ;;
   esac
 
-  got_cross_cc=no
-  unset build_static
-
-  if test -n "$target_cc"; then
-      write_c_skeleton
-      if ! do_compiler "$target_cc" $target_cflags \
-           -o $TMPE $TMPC -static ; then
-          # For host systems we might get away with building without -static
-          if do_compiler "$target_cc" $target_cflags \
-                         -o $TMPE $TMPC ; then
-              got_cross_cc=yes
-          fi
-      else
-          got_cross_cc=yes
-          build_static=y
-      fi
-  elif test -n "$target_as" && test -n "$target_ld"; then
-      # Special handling for assembler only tests
-      case $target in
-          tricore-softmmu) got_cross_cc=yes ;;
-      esac
-  fi
-
   if test $got_cross_cc = yes; then
       # Test for compiler features for optional tests. We only do this
       # for cross compilers because ensuring the docker containers based
-- 
2.36.1

Re: [PATCH 2/2] build: try both native and cross compilers for linux-user tests

[Matheus Kowalczuk Ferst]

On 22/06/2022 10:47, Paolo Bonzini wrote:
> Configure is trying to fall back on cross compilers for targets that
> can have bi-arch or bi-endian toolchains, but there are many corner
> cases where just checking the name can go wrong.  For example, the RHEL
> ppc64le compiler is bi-arch and bi-endian, but multilibs are disabled.
> Therefore it cannot be used to build 32-bit hosted binaries like the
> linux-user TCG tests.
> 
> Trying the cross compiler first also does not work, and an example for
> this is also ppc64le.  The powerpc64-linux-gnu-gcc binary from the
> cross-gcc package is theoretically multilib-friendly, but it cannot
> find the CRT files on a ppc64le host, because they are not in the .../le
> multilib subdirectory.
> 
> This can be fixed by testing both the native compiler and the cross
> compiler, and proceeding with the first one that works.  To do this,
> move the compiler usability check from the tests/tcg snippet to inside
> probe_target_compiler and, while at it, restrict it to just the user-mode
> emulation tests; if a compiler is not able to build nostdlib freestanding
> binaries the installation is broken.
> ---
>   configure | 166 ++++++++++++++++++++++++++++++++----------------------
>   1 file changed, 100 insertions(+), 66 deletions(-)
> 

With this patch I can successfully build ppc64le-linux-user tests on 
ppc64le hosts (Debian and Fedora) and ppc{,64}-linux-user tests on ppc64 
hosts (Void Linux).

Tested-by: Matheus Ferst <matheus.ferst@eldorado.org.br>

Thanks,
Matheus K. Ferst
Instituto de Pesquisas ELDORADO <http://www.eldorado.org.br/>
Analista de Software
Aviso Legal - Disclaimer <https://www.eldorado.org.br/disclaimer.html>