[Qemu-devel] [PULL for 3.0 00/17] docker fixes and tcg test tweak

[Qemu-devel] [PULL for 3.0 00/17] docker fixes and tcg test tweak

[Alex Bennée]

The following changes since commit 768cef2974fb1fa30dd188b043ea737e13fea477:

  Merge remote-tracking branch 'remotes/ehabkost/tags/x86-next-pull-request' into staging (2018-07-24 10:37:52 +0100)

are available in the Git repository at:

  https://github.com/stsquad/qemu.git tags/pull-docker-fixes-for-3.0-240718-1

for you to fetch changes up to 1a3bdc6111482fbb41eacacf850002e4589a81ef:

  tests/tcg: remove runcom test (2018-07-24 11:45:25 +0100)

----------------------------------------------------------------
docker fixes & tcg test tweak

  - graceful handling of testing under cross-compile
  - fixes for debootstrap handling
  - more helpful errors (binfmt_misc/EXECUTABLE missing)
  - drop runcom TCG test

----------------------------------------------------------------
Alex Bennée (16):
      tests/.gitignore: don't ignore docker tests
      docker: base debian-tricore on qemu:debian9
      docker: par down QEMU_CONFIGURE_OPTS in debian-tricore-cross
      docker: fail more gracefully on docker.py check
      docker: split configure_qemu from build_qemu
      docker: move make check into check_qemu helper
      docker: gracefully skip check_qemu
      docker: Makefile.include don't include partial images
      docker: add test-unit runner
      docker: add expansion for docker-test-FOO to Makefile.include
      docker: drop QEMU_TARGET check, fallback in EXECUTABLE not set
      docker: report hint when docker.py check fails
      docker: add commentary to debian-bootstrap.docker
      docker: ignore distro versioning of debootstrap
      docker: perform basic binfmt_misc validation in docker.py
      tests/tcg: remove runcom test

Philippe Mathieu-Daudé (1):
      docker: Update debootstrap script after Debian migration from Alioth to Salsa

 tests/.gitignore                                   |   1 +
 tests/docker/Makefile.include                      |  23 ++-
 tests/docker/common.rc                             |  24 ++-
 tests/docker/docker.py                             |  37 +++-
 tests/docker/dockerfiles/debian-bootstrap.docker   |   1 +
 tests/docker/dockerfiles/debian-bootstrap.pre      |  13 +-
 .../docker/dockerfiles/debian-tricore-cross.docker |   6 +-
 tests/docker/test-clang                            |   2 +-
 tests/docker/test-debug                            |   2 +-
 tests/docker/test-full                             |   2 +-
 tests/docker/test-quick                            |   2 +-
 tests/docker/test-unit                             |  21 +++
 tests/tcg/i386/Makefile.target                     |   5 -
 tests/tcg/i386/README                              |   3 -
 tests/tcg/i386/pi_10.com                           | Bin 54 -> 0 bytes
 tests/tcg/i386/runcom.c                            | 192 ---------------------
 16 files changed, 111 insertions(+), 223 deletions(-)
 create mode 100755 tests/docker/test-unit
 delete mode 100644 tests/tcg/i386/pi_10.com
 delete mode 100644 tests/tcg/i386/runcom.c

-- 
2.17.1

[Qemu-devel] [PULL 01/17] tests/.gitignore: don't ignore docker tests

[Alex Bennée]

The .gitignore was being a little over enthusiastic hiding files.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/.gitignore b/tests/.gitignore
index 08e2df1ce1..72c18aaab0 100644
--- a/tests/.gitignore
+++ b/tests/.gitignore
@@ -9,6 +9,7 @@ qht-bench
 rcutorture
 test-*
 !test-*.c
+!docker/test-*
 test-qapi-commands.[ch]
 test-qapi-events.[ch]
 test-qapi-types.[ch]
-- 
2.17.1

[Qemu-devel] [PULL 02/17] docker: base debian-tricore on qemu:debian9

[Alex Bennée]

We need both git and a working compiler to build the tools. Although
the qemu:debian9 image also has a bunch of extra dependencies it would
be fairly unusual for a user not to already have this layer available
for one of our many other docker images so lets not complicate things.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker
index 898b8dd511..180ca646c8 100644
--- a/tests/docker/dockerfiles/debian-tricore-cross.docker
+++ b/tests/docker/dockerfiles/debian-tricore-cross.docker
@@ -7,7 +7,7 @@
 #
 # SPDX-License-Identifier: GPL-2.0-or-later
 #
-FROM debian:9
+FROM qemu:debian9
 
 MAINTAINER Philippe Mathieu-Daudé <f4bug@amsat.org>
 
-- 
2.17.1

[Qemu-devel] [PULL 03/17] docker: par down QEMU_CONFIGURE_OPTS in debian-tricore-cross

[Alex Bennée]

This image isn't going to build anything significant as it is just
intended for building test cases. In case it does end up getting
inadvertently included in a build lets aim for the minimal possible
product.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker
index 180ca646c8..4a0f7706a3 100644
--- a/tests/docker/dockerfiles/debian-tricore-cross.docker
+++ b/tests/docker/dockerfiles/debian-tricore-cross.docker
@@ -19,5 +19,5 @@ RUN git clone --single-branch \
     make && make install && \
     rm -rf /usr/src/binutils
 
-# Specify the cross prefix for this image (see tests/docker/common.rc)
-ENV QEMU_CONFIGURE_OPTS --cross-prefix=tricore-
+# This image isn't designed for building QEMU but building tests
+ENV QEMU_CONFIGURE_OPTS --disable-system --disable-user
-- 
2.17.1

[Qemu-devel] [PULL 04/17] docker: fail more gracefully on docker.py check

[Alex Bennée]

As this is called directly from the Makefile while determining
dependencies and it is possible the user was configured in one window
but not have credentials in the other. Let's catch the Exceptions and
deal with it quietly.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reported-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 69e7130db7..2f81c6b13b 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -479,7 +479,12 @@ class CheckCommand(SubCommand):
     def run(self, args, argv):
         tag = args.tag
 
-        dkr = Docker()
+        try:
+            dkr = Docker()
+        except:
+            print("Docker not set up")
+            return 1
+
         info = dkr.inspect_tag(tag)
         if info is None:
             print("Image does not exist")
-- 
2.17.1

[Qemu-devel] [PULL 05/17] docker: split configure_qemu from build_qemu

[Alex Bennée]

This allows some tests that just want to configure QEMU's source tree
to do so.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/common.rc b/tests/docker/common.rc
index 046f8a5921..ba1f942328 100755
--- a/tests/docker/common.rc
+++ b/tests/docker/common.rc
@@ -21,7 +21,7 @@ requires()
     done
 }
 
-build_qemu()
+configure_qemu()
 {
     config_opts="--enable-werror \
                  ${TARGET_LIST:+--target-list=${TARGET_LIST}} \
@@ -32,6 +32,11 @@ build_qemu()
     echo $config_opts
     $QEMU_SRC/configure $config_opts || \
         { cat config.log && test_fail "Failed to run 'configure'"; }
+}
+
+build_qemu()
+{
+    configure_qemu $@
     make $MAKEFLAGS
 }
 
-- 
2.17.1

[Qemu-devel] [PULL 06/17] docker: move make check into check_qemu helper

[Alex Bennée]

Not all docker images can run the check step. Let's move everything
into a common helper so we don't need to replicate checks in the
future.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/common.rc b/tests/docker/common.rc
index ba1f942328..4ff5974016 100755
--- a/tests/docker/common.rc
+++ b/tests/docker/common.rc
@@ -40,6 +40,17 @@ build_qemu()
     make $MAKEFLAGS
 }
 
+check_qemu()
+{
+    # default to make check unless the caller specifies
+    if test -z "$@"; then
+        INVOCATION="check"
+    else
+        INVOCATION="$@"
+    fi
+    make $MAKEFLAGS $INVOCATION
+}
+
 test_fail()
 {
     echo "$@"
diff --git a/tests/docker/test-clang b/tests/docker/test-clang
index e90a793178..324e341cea 100755
--- a/tests/docker/test-clang
+++ b/tests/docker/test-clang
@@ -23,5 +23,5 @@ OPTS="--cxx=clang++ --cc=clang --host-cc=clang"
 #OPTS="$OPTS --extra-cflags=-fsanitize=undefined \
     #--extra-cflags=-fno-sanitize=float-divide-by-zero"
 build_qemu $OPTS
-make $MAKEFLAGS check
+check_qemu
 install_qemu
diff --git a/tests/docker/test-debug b/tests/docker/test-debug
index d3f9f70d01..137f4f2ddc 100755
--- a/tests/docker/test-debug
+++ b/tests/docker/test-debug
@@ -22,5 +22,5 @@ OPTS="--cxx=clang++ --cc=clang --host-cc=clang"
 OPTS="--enable-debug --enable-sanitizers $OPTS"
 
 build_qemu $OPTS
-make $MAKEFLAGS V=1 check
+check_qemu check V=1
 install_qemu
diff --git a/tests/docker/test-full b/tests/docker/test-full
index b4e42d25d7..aadc0f00a2 100755
--- a/tests/docker/test-full
+++ b/tests/docker/test-full
@@ -15,4 +15,4 @@
 
 cd "$BUILD_DIR"
 
-build_qemu && make check $MAKEFLAGS && install_qemu
+build_qemu && check_qemu && install_qemu
diff --git a/tests/docker/test-quick b/tests/docker/test-quick
index 3b7bce6105..eee59c55fb 100755
--- a/tests/docker/test-quick
+++ b/tests/docker/test-quick
@@ -18,5 +18,5 @@ cd "$BUILD_DIR"
 DEF_TARGET_LIST="x86_64-softmmu,aarch64-softmmu"
 TARGET_LIST=${TARGET_LIST:-$DEF_TARGET_LIST} \
 build_qemu
-make check $MAKEFLAGS
+check_qemu
 install_qemu
-- 
2.17.1

Re: [PULL 06/17] docker: move make check into check_qemu helper

[Peter Maydell]

On Tue, 24 Jul 2018 at 11:51, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Not all docker images can run the check step. Let's move everything
> into a common helper so we don't need to replicate checks in the
> future.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>
> diff --git a/tests/docker/common.rc b/tests/docker/common.rc
> index ba1f942328..4ff5974016 100755
> --- a/tests/docker/common.rc
> +++ b/tests/docker/common.rc
> @@ -40,6 +40,17 @@ build_qemu()
>      make $MAKEFLAGS
>  }
>
> +check_qemu()
> +{
> +    # default to make check unless the caller specifies
> +    if test -z "$@"; then
> +        INVOCATION="check"
> +    else
> +        INVOCATION="$@"
> +    fi

Philippe pointed out on IRC that a recent change has resulted
in the shell now pointing out an error in the above code
(with "common.rc: line 50: test: check: binary operator expected").

test -z wants a single string argument, so passing it "$@" is wrong
(that will expand to multiple arguments if the function was
called with multiple arguments).

Probably what is wanted is
  if [ $# == 0 ]; then

ie test whether the function was passed no arguments, rather
than looking specifically for whether all its arguments
squashed together end up being the empty string. (If you
really want the latter that's "$*".)

> +    make $MAKEFLAGS $INVOCATION
> +}

thanks
-- PMM

[Qemu-devel] [PULL 07/17] docker: gracefully skip check_qemu

[Alex Bennée]

Not all our images are able to run the tests. Rather than use features
we can just check for the existence and run-ability of gtester. If the
image has been setup for binfmt_misc it will be able to run anyway.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/common.rc b/tests/docker/common.rc
index 4ff5974016..4011561587 100755
--- a/tests/docker/common.rc
+++ b/tests/docker/common.rc
@@ -48,7 +48,13 @@ check_qemu()
     else
         INVOCATION="$@"
     fi
-    make $MAKEFLAGS $INVOCATION
+
+    if command -v gtester > /dev/null 2>&1 && \
+           gtester --version > /dev/null 2>&1; then
+        make $MAKEFLAGS $INVOCATION
+    else
+        echo "No working gtester, skipping make $INVOCATION"
+    fi
 }
 
 test_fail()
-- 
2.17.1

[Qemu-devel] [PULL 08/17] docker: Makefile.include don't include partial images

[Alex Bennée]

Rename DOCKER_INTERMEDIATE_IMAGES to DOCKER_PARTIAL_IMAGES and add the
incomplete cross compiler images that can build tests but can't build
QEMU itself. We also add debian, debian-bootstrap and the tricode
images to the list.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index b2a7e761cc..09fb7db7fa 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -6,7 +6,7 @@ DOCKER_SUFFIX := .docker
 DOCKER_FILES_DIR := $(SRC_PATH)/tests/docker/dockerfiles
 DOCKER_DEPRECATED_IMAGES := debian
 # we don't run tests on intermediate images (used as base by another image)
-DOCKER_INTERMEDIATE_IMAGES := debian8 debian9 debian8-mxe debian-ports debian-sid
+DOCKER_PARTIAL_IMAGES := debian debian8 debian9 debian8-mxe debian-ports debian-sid debian-bootstrap
 DOCKER_IMAGES := $(filter-out $(DOCKER_DEPRECATED_IMAGES),$(sort $(notdir $(basename $(wildcard $(DOCKER_FILES_DIR)/*.docker)))))
 DOCKER_TARGETS := $(patsubst %,docker-image-%,$(DOCKER_IMAGES))
 # Use a global constant ccache directory to speed up repetitive builds
@@ -121,6 +121,11 @@ docker-image-travis: NOUSER=1
 # Specialist build images, sometimes very limited tools
 docker-image-tricore-cross: docker-image-debian9
 
+# These images may be good enough for building tests but not for test builds
+DOCKER_PARTIAL_IMAGES += debian-alpha-cross debian-hppa-cross debian-m68k-cross debian-sh4-cross
+DOCKER_PARTIAL_IMAGES += debian-sparc64-cross debian-mips64-cross debian-riscv64-cross
+DOCKER_PARTIAL_IMAGES += debian-tricore-cross debian-powerpc-cross fedora-i386-cross
+
 # Rules for building linux-user powered images
 #
 # These are slower than using native cross compiler setups but can
@@ -137,7 +142,7 @@ docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
 DOCKER_USER_IMAGES += debian-powerpc-user
 
 # Expand all the pre-requistes for each docker image and test combination
-$(foreach i,$(filter-out $(DOCKER_INTERMEDIATE_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
+$(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPRECATED_IMAGES)), \
 	$(foreach t,$(DOCKER_TESTS) $(DOCKER_TOOLS), \
 		$(eval .PHONY: docker-$t@$i) \
 		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
-- 
2.17.1

[Qemu-devel] [PULL 09/17] docker: add test-unit runner

[Alex Bennée]

This test doesn't even build QEMU, it just builds and runs all the
unit tests. Intended to make checking unit tests on all docker images
easier.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/test-unit b/tests/docker/test-unit
new file mode 100755
index 0000000000..8905d01150
--- /dev/null
+++ b/tests/docker/test-unit
@@ -0,0 +1,21 @@
+#!/bin/bash -e
+#
+# Build and run the unit tests
+#
+# Copyright (c) 2018 Linaro Ltd.
+#
+# Authors:
+#  Alex Bennée <alex.bennee@linaro.org>
+#
+# This work is licensed under the terms of the GNU GPL, version 2
+# or (at your option) any later version. See the COPYING file in
+# the top-level directory.
+
+. common.rc
+
+cd "$BUILD_DIR"
+
+# although we are not building QEMU itself we still need a configured
+# build for the unit tests to be built and run
+configure_qemu
+check_qemu check-unit
-- 
2.17.1

[Qemu-devel] [PULL 10/17] docker: add expansion for docker-test-FOO to Makefile.include

[Alex Bennée]

This allows us to run a particular test on all docker images. For
example:

  make docker-test-unit

Will run the unit tests on every supported image. At the same time
rename docker-test to docker-all-tests to be clearer.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 09fb7db7fa..8fbb076396 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -148,7 +148,8 @@ $(foreach i,$(filter-out $(DOCKER_PARTIAL_IMAGES),$(DOCKER_IMAGES) $(DOCKER_DEPR
 		$(eval docker-$t@$i: docker-image-$i docker-run-$t@$i) \
 	) \
 	$(foreach t,$(DOCKER_TESTS), \
-		$(eval docker-test: docker-$t@$i) \
+		$(eval docker-all-tests: docker-$t@$i) \
+		$(eval docker-$t: docker-$t@$i) \
 	) \
 )
 
@@ -158,7 +159,8 @@ docker:
 	@echo 'Available targets:'
 	@echo
 	@echo '    docker:              Print this help.'
-	@echo '    docker-test:         Run all image/test combinations.'
+	@echo '    docker-all-tests:    Run all image/test combinations.'
+	@echo '    docker-TEST:         Run TEST on all image combinations.'
 	@echo '    docker-clean:        Kill and remove residual docker testing containers.'
 	@echo '    docker-TEST@IMAGE:   Run "TEST" in container "IMAGE".'
 	@echo '                         Note: "TEST" is one of the listed test name,'
-- 
2.17.1

[Qemu-devel] [PULL 11/17] docker: drop QEMU_TARGET check, fallback in EXECUTABLE not set

[Alex Bennée]

The addition of QEMU_TARGET was intended to ensure we fall back to
checking for the existence of an image if the build system was not
currently configured to build it. However this breaks the direct use
of the rule for building custom binfmt_misc images. We already check
for EXECUTABLE so let us just use that as a proxy for deciding if we
are just going to check the image exits.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 8fbb076396..05afeb64a7 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -58,13 +58,11 @@ docker-image-%: $(DOCKER_FILES_DIR)/%.docker
 docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
 	$(if $(EXECUTABLE),,\
 		$(error EXECUTABLE not set, debootstrap of debian-$* would fail))
-	$(if $(wildcard $(EXECUTABLE)),,\
-		$(error Please build $(EXECUTABLE) first))
 	$(if $(DEB_ARCH),,\
 		$(error DEB_ARCH not set, debootstrap of debian-$* would fail))
 	$(if $(DEB_TYPE),,\
 		$(error DEB_TYPE not set, debootstrap of debian-$* would fail))
-	$(if $(filter $(QEMU_TARGET),$(TARGET_DIRS)),				\
+	$(if $(wildcard $(EXECUTABLE)),						\
 		$(call quiet-command,						\
 			DEB_ARCH=$(DEB_ARCH)					\
 			DEB_TYPE=$(DEB_TYPE)					\
@@ -136,7 +134,6 @@ DOCKER_PARTIAL_IMAGES += debian-tricore-cross debian-powerpc-cross fedora-i386-c
 # broken so we need a qemu-linux-user for this target
 docker-binfmt-image-debian-powerpc-user: DEB_ARCH = powerpc
 docker-binfmt-image-debian-powerpc-user: DEB_TYPE = jessie
-docker-binfmt-image-debian-powerpc-user: QEMU_TARGET = ppc-linux-user
 docker-binfmt-image-debian-powerpc-user: EXECUTABLE = ${BUILD_DIR}/ppc-linux-user/qemu-ppc
 docker-image-debian-powerpc-user-cross: docker-binfmt-image-debian-powerpc-user
 DOCKER_USER_IMAGES += debian-powerpc-user
-- 
2.17.1

[Qemu-devel] [PULL 12/17] docker: report hint when docker.py check fails

[Alex Bennée]

When a check fails we currently just report why we failed. This is not
totally helpful to people who want to boot-strap a new image. Report a
hint as to why it failed.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Suggested-by: Fam Zheng <famz@redhat.com>

diff --git a/tests/docker/Makefile.include b/tests/docker/Makefile.include
index 05afeb64a7..1aaa795743 100644
--- a/tests/docker/Makefile.include
+++ b/tests/docker/Makefile.include
@@ -73,7 +73,8 @@ docker-binfmt-image-debian-%: $(DOCKER_FILES_DIR)/debian-bootstrap.docker
 			$(if $(EXECUTABLE),--include-executable=$(EXECUTABLE)), \
 			"BUILD","binfmt debian-$* (debootstrapped)"),		\
 		$(call quiet-command,						\
-			$(DOCKER_SCRIPT) check --quiet qemu:debian-$* $<,	\
+			$(DOCKER_SCRIPT) check --quiet qemu:debian-$* $< || 	\
+			{ echo "You will need to build $(EXECUTABLE)"; exit 1;},\
 			"CHECK", "debian-$* exists"))
 
 endif
-- 
2.17.1

[Qemu-devel] [PULL 13/17] docker: Update debootstrap script after Debian migration from Alioth to Salsa

[Alex Bennée]

From: Philippe Mathieu-Daudé <f4bug@amsat.org>

This silents the following warning:

  Cloning into './debootstrap.git'...
  warning: redirecting to https://salsa.debian.org/installer-team/debootstrap.git/

See https://lists.debian.org/debian-devel-announce/2018/01/msg00004.html

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
index 56e1aa7a21..ea324d6e4a 100755
--- a/tests/docker/dockerfiles/debian-bootstrap.pre
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -62,7 +62,7 @@ if [ -z $DEBOOTSTRAP_DIR ]; then
         NEED_DEBOOTSTRAP=true
     fi
     if $NEED_DEBOOTSTRAP; then
-        DEBOOTSTRAP_SOURCE=https://anonscm.debian.org/git/d-i/debootstrap.git
+        DEBOOTSTRAP_SOURCE=https://salsa.debian.org/installer-team/debootstrap.git
         git clone ${DEBOOTSTRAP_SOURCE} ./debootstrap.git
         export DEBOOTSTRAP_DIR=./debootstrap.git
         DEBOOTSTRAP=./debootstrap.git/debootstrap
-- 
2.17.1

[Qemu-devel] [PULL 14/17] docker: add commentary to debian-bootstrap.docker

[Alex Bennée]

This is just a note that later versions of debootstrap don't
technically need this hack.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/dockerfiles/debian-bootstrap.docker b/tests/docker/dockerfiles/debian-bootstrap.docker
index 14212b9cf4..e13c26a7ed 100644
--- a/tests/docker/dockerfiles/debian-bootstrap.docker
+++ b/tests/docker/dockerfiles/debian-bootstrap.docker
@@ -9,6 +9,7 @@ FROM scratch
 ADD . /
 
 # Patch all mounts as docker already has stuff set up
+# (this is not needed for later debootstraps but is harmless atm)
 RUN sed -i 's/in_target mount/echo not for docker in_target mount/g' /debootstrap/functions
 
 # Run stage 2
-- 
2.17.1

[Qemu-devel] [PULL 15/17] docker: ignore distro versioning of debootstrap

[Alex Bennée]

We do a minimum version check for the debootstrap but if the distro
has added their own minor version tick it would fail and fall-back to
the SCM version. This is sub-optimal as the latest/greatest version
may be broken at any one particular time. We fix that with a little
sed magic on the version string before passing to our ugly shell
versioning check.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

diff --git a/tests/docker/dockerfiles/debian-bootstrap.pre b/tests/docker/dockerfiles/debian-bootstrap.pre
index ea324d6e4a..3b0ef95374 100755
--- a/tests/docker/dockerfiles/debian-bootstrap.pre
+++ b/tests/docker/dockerfiles/debian-bootstrap.pre
@@ -56,10 +56,13 @@ if [ -z $DEBOOTSTRAP_DIR ]; then
     if [ -z $DEBOOTSTRAP ]; then
         echo "No debootstrap installed, attempting to install from SCM"
         NEED_DEBOOTSTRAP=true
-    elif ! (echo "${MIN_DEBOOTSTRAP_VERSION}" ; "${DEBOOTSTRAP}" --version \
-            | cut -d ' ' -f 2) | sort -t . -n -k 1,1 -k 2,2 -k 3,3 -c &>/dev/null; then
-        echo "debootstrap too old, attempting to install from SCM"
-        NEED_DEBOOTSTRAP=true
+    else
+        INSTALLED_VERSION=$(${DEBOOTSTRAP} --version | sed 's/debootstrap \([0-9\.]*\)[^0-9\.]*.*/\1/')
+        if ! (echo "${MIN_DEBOOTSTRAP_VERSION}" ; echo "${INSTALLED_VERSION}") \
+                | sort -t . -n -k 1,1 -k 2,2 -k 3,3 -C ; then
+            echo "debootstrap too old, attempting to install from SCM"
+            NEED_DEBOOTSTRAP=true
+        fi
     fi
     if $NEED_DEBOOTSTRAP; then
         DEBOOTSTRAP_SOURCE=https://salsa.debian.org/installer-team/debootstrap.git
-- 
2.17.1

[Qemu-devel] [PULL 16/17] docker: perform basic binfmt_misc validation in docker.py

[Alex Bennée]

Setting up binfmt_misc is outside of the scope of the docker.py script
but we can at least validate it with any given executable so we have a
more useful error message than the sed line of deboostrap failing
cryptically.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reported-by: Richard Henderson <richard.henderson@linaro.org>

diff --git a/tests/docker/docker.py b/tests/docker/docker.py
index 2f81c6b13b..d3006d4dae 100755
--- a/tests/docker/docker.py
+++ b/tests/docker/docker.py
@@ -112,6 +112,31 @@ def _copy_binary_with_libs(src, dest_dir):
             so_path = os.path.dirname(l)
             _copy_with_mkdir(l , dest_dir, so_path)
 
+
+def _check_binfmt_misc(executable):
+    """Check binfmt_misc has entry for executable in the right place.
+
+    The details of setting up binfmt_misc are outside the scope of
+    this script but we should at least fail early with a useful
+    message if it won't work."""
+
+    binary = os.path.basename(executable)
+    binfmt_entry = "/proc/sys/fs/binfmt_misc/%s" % (binary)
+
+    if not os.path.exists(binfmt_entry):
+        print ("No binfmt_misc entry for %s" % (binary))
+        return False
+
+    with open(binfmt_entry) as x: entry = x.read()
+
+    qpath = "/usr/bin/%s" % (binary)
+    if not re.search("interpreter %s\n" % (qpath), entry):
+        print ("binfmt_misc for %s does not point to %s" % (binary, qpath))
+        return False
+
+    return True
+
+
 def _read_qemu_dockerfile(img_name):
     # special case for Debian linux-user images
     if img_name.startswith("debian") and img_name.endswith("user"):
@@ -315,6 +340,11 @@ class BuildCommand(SubCommand):
             # Create a docker context directory for the build
             docker_dir = tempfile.mkdtemp(prefix="docker_build")
 
+            # Validate binfmt_misc will work
+            if args.include_executable:
+                if not _check_binfmt_misc(args.include_executable):
+                    return 1
+
             # Is there a .pre file to run in the build context?
             docker_pre = os.path.splitext(args.dockerfile)[0]+".pre"
             if os.path.exists(docker_pre):
-- 
2.17.1

[Qemu-devel] [PULL 17/17] tests/tcg: remove runcom test

[Alex Bennée]

The combination of being rather esoteric and needing to support mmap @
0 means this only ever worked under translation. It has now regressed
even further and is no longer useful. Kill it.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>

diff --git a/tests/tcg/i386/Makefile.target b/tests/tcg/i386/Makefile.target
index c1997a1624..b4033ba3d1 100644
--- a/tests/tcg/i386/Makefile.target
+++ b/tests/tcg/i386/Makefile.target
@@ -29,11 +29,6 @@ test-i386: test-i386.c test-i386-code16.S test-i386-vm86.S test-i386.h test-i386
 	$(CC) $(CFLAGS) $(LDFLAGS) -o $@ \
 	   $(<D)/test-i386.c $(<D)/test-i386-code16.S $(<D)/test-i386-vm86.S -lm
 
-# Specialist test runners
-run-runcom: TIMEOUT=30
-run-runcom: runcom pi_10.com
-	$(call run-test,$<,$(QEMU) ./runcom $(I386_SRC)/pi_10.com,"$< on $(TARGET_NAME)")
-
 ifeq ($(SPEED), slow)
 
 test-i386-fprem.ref: test-i386-fprem
diff --git a/tests/tcg/i386/README b/tests/tcg/i386/README
index 7a0a47bf27..09e88f30dc 100644
--- a/tests/tcg/i386/README
+++ b/tests/tcg/i386/README
@@ -25,9 +25,6 @@ and host CPUs.
 test-i386-fprem
 ---------------
 
-runcom
-------
-
 test-mmap
 ---------
 
diff --git a/tests/tcg/i386/pi_10.com b/tests/tcg/i386/pi_10.com
deleted file mode 100644
index 8993ba1a51bbc2aaddc3aa99ceea37ab3bf9ea9e..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001

literal 54
zcmV-60LlM|6SaSi!9cr&CiAO|!E5OPMfdfRtCIKOQkA$004Db(1I1kv)#B#&Gz2h`
MtF!{mAxm}F!x~T=Hvj+t

diff --git a/tests/tcg/i386/runcom.c b/tests/tcg/i386/runcom.c
deleted file mode 100644
index d60342bfc6..0000000000
--- a/tests/tcg/i386/runcom.c
+++ /dev/null
@@ -1,192 +0,0 @@
-/*
- * Simple example of use of vm86: launch a basic .com DOS executable
- */
-#include <stdlib.h>
-#include <stdio.h>
-#include <string.h>
-#include <inttypes.h>
-#include <unistd.h>
-#include <fcntl.h>
-#include <sys/mman.h>
-#include <signal.h>
-
-#include <linux/unistd.h>
-#include <asm/vm86.h>
-
-extern int vm86 (unsigned long int subfunction,
-		 struct vm86plus_struct *info);
-
-#define VIF_MASK                0x00080000
-
-//#define SIGTEST
-
-#define COM_BASE_ADDR    0x10100
-
-static void usage(void)
-{
-    printf("runcom version 0.1 (c) 2003 Fabrice Bellard\n"
-           "usage: runcom file.com\n"
-           "VM86 Run simple .com DOS executables (linux vm86 test mode)\n");
-    exit(1);
-}
-
-static inline void set_bit(uint8_t *a, unsigned int bit)
-{
-    a[bit / 8] |= (1 << (bit % 8));
-}
-
-static inline uint8_t *seg_to_linear(unsigned int seg, unsigned int reg)
-{
-    return (uint8_t *)((seg << 4) + (reg & 0xffff));
-}
-
-static inline void pushw(struct vm86_regs *r, int val)
-{
-    r->esp = (r->esp & ~0xffff) | ((r->esp - 2) & 0xffff);
-    *(uint16_t *)seg_to_linear(r->ss, r->esp) = val;
-}
-
-void dump_regs(struct vm86_regs *r)
-{
-    fprintf(stderr,
-            "EAX=%08lx EBX=%08lx ECX=%08lx EDX=%08lx\n"
-            "ESI=%08lx EDI=%08lx EBP=%08lx ESP=%08lx\n"
-            "EIP=%08lx EFL=%08lx\n"
-            "CS=%04x DS=%04x ES=%04x SS=%04x FS=%04x GS=%04x\n",
-            r->eax, r->ebx, r->ecx, r->edx, r->esi, r->edi, r->ebp, r->esp,
-            r->eip, r->eflags,
-            r->cs, r->ds, r->es, r->ss, r->fs, r->gs);
-}
-
-#ifdef SIGTEST
-void alarm_handler(int sig)
-{
-    fprintf(stderr, "alarm signal=%d\n", sig);
-    alarm(1);
-}
-#endif
-
-int main(int argc, char **argv)
-{
-    uint8_t *vm86_mem;
-    const char *filename;
-    int fd, ret, seg;
-    struct vm86plus_struct ctx;
-    struct vm86_regs *r;
-
-    if (argc != 2)
-        usage();
-    filename = argv[1];
-
-    vm86_mem = mmap((void *)0x00000000, 0x110000,
-                    PROT_WRITE | PROT_READ | PROT_EXEC,
-                    MAP_FIXED | MAP_ANON | MAP_PRIVATE, -1, 0);
-    if (vm86_mem == MAP_FAILED) {
-        perror("mmap");
-        exit(1);
-    }
-#ifdef SIGTEST
-    {
-        struct sigaction act;
-
-        act.sa_handler = alarm_handler;
-        sigemptyset(&act.sa_mask);
-        act.sa_flags = 0;
-        sigaction(SIGALRM, &act, NULL);
-        alarm(1);
-    }
-#endif
-
-    /* load the MSDOS .com executable */
-    fd = open(filename, O_RDONLY);
-    if (fd < 0) {
-        perror(filename);
-        exit(1);
-    }
-    ret = read(fd, vm86_mem + COM_BASE_ADDR, 65536 - 256);
-    if (ret < 0) {
-        perror("read");
-        exit(1);
-    }
-    close(fd);
-
-    memset(&ctx, 0, sizeof(ctx));
-    /* init basic registers */
-    r = &ctx.regs;
-    r->eip = 0x100;
-    r->esp = 0xfffe;
-    seg = (COM_BASE_ADDR - 0x100) >> 4;
-    r->cs = seg;
-    r->ss = seg;
-    r->ds = seg;
-    r->es = seg;
-    r->fs = seg;
-    r->gs = seg;
-    r->eflags = VIF_MASK;
-
-    /* put return code */
-    set_bit((uint8_t *)&ctx.int_revectored, 0x21);
-    *seg_to_linear(r->cs, 0) = 0xb4; /* mov ah, $0 */
-    *seg_to_linear(r->cs, 1) = 0x00;
-    *seg_to_linear(r->cs, 2) = 0xcd; /* int $0x21 */
-    *seg_to_linear(r->cs, 3) = 0x21;
-    pushw(&ctx.regs, 0x0000);
-
-    /* the value of these registers seem to be assumed by pi_10.com */
-    r->esi = 0x100;
-    r->ecx = 0xff;
-    r->ebp = 0x0900;
-    r->edi = 0xfffe;
-
-    for(;;) {
-        ret = vm86(VM86_ENTER, &ctx);
-        switch(VM86_TYPE(ret)) {
-        case VM86_INTx:
-            {
-                int int_num, ah;
-
-                int_num = VM86_ARG(ret);
-                if (int_num != 0x21)
-                    goto unknown_int;
-                ah = (r->eax >> 8) & 0xff;
-                switch(ah) {
-                case 0x00: /* exit */
-                    exit(0);
-                case 0x02: /* write char */
-                    {
-                        uint8_t c = r->edx;
-                        write(1, &c, 1);
-                    }
-                    break;
-                case 0x09: /* write string */
-                    {
-                        uint8_t c;
-                        for(;;) {
-                            c = *seg_to_linear(r->ds, r->edx);
-                            if (c == '$')
-                                break;
-                            write(1, &c, 1);
-                        }
-                        r->eax = (r->eax & ~0xff) | '$';
-                    }
-                    break;
-                default:
-                unknown_int:
-                    fprintf(stderr, "unsupported int 0x%02x\n", int_num);
-                    dump_regs(&ctx.regs);
-                    //                    exit(1);
-                }
-            }
-            break;
-        case VM86_SIGNAL:
-            /* a signal came, we just ignore that */
-            break;
-        case VM86_STI:
-            break;
-        default:
-            fprintf(stderr, "unhandled vm86 return code (0x%x)\n", ret);
-            dump_regs(&ctx.regs);
-            exit(1);
-        }
-    }
-}
-- 
2.17.1

Re: [Qemu-devel] [PULL for 3.0 00/17] docker fixes and tcg test tweak

[Peter Maydell]

On 24 July 2018 at 11:50, Alex Bennée <alex.bennee@linaro.org> wrote:
> The following changes since commit 768cef2974fb1fa30dd188b043ea737e13fea477:
>
>   Merge remote-tracking branch 'remotes/ehabkost/tags/x86-next-pull-request' into staging (2018-07-24 10:37:52 +0100)
>
> are available in the Git repository at:
>
>   https://github.com/stsquad/qemu.git tags/pull-docker-fixes-for-3.0-240718-1
>
> for you to fetch changes up to 1a3bdc6111482fbb41eacacf850002e4589a81ef:
>
>   tests/tcg: remove runcom test (2018-07-24 11:45:25 +0100)
>
> ----------------------------------------------------------------
> docker fixes & tcg test tweak
>
>   - graceful handling of testing under cross-compile
>   - fixes for debootstrap handling
>   - more helpful errors (binfmt_misc/EXECUTABLE missing)
>   - drop runcom TCG test
>

Applied, thanks.

-- PMM