[Qemu-devel] [PATCH 0/3] Add a CentOS test image to run docker tests

[Qemu-devel] [PATCH 0/3] Add a CentOS test image to run docker tests

[Fam Zheng]

Docker testing on patchew has long suffered from 'make check' hangings. The
cleanness of VM testing is the cure. Now let's add a CentOS 7 image to run the
tests.  It's purely ad-hoc, but hopefully still easy to understand and use for
everyone.

The first patch makes passing source code from host to the container in VM
working, and is a nice clean up.

The second patch makes caches work, to speed up repetitive runs like on
patchew.

The last patch adds the new image that does the job. Two out of three docker
tests running on patchew.org are added to the image. I'll wait for Peter to fix
the 'docker-test-quick@centos6' hanging (oob test) before adding it too.

Fam

Fam Zheng (3):
  archive-source.sh: Drop submodule code
  tests: Add an option for snapshot (default: off)
  tests: Add centos VM testing

 scripts/archive-source.sh | 47 +++------------------------
 tests/vm/basevm.py        |  7 +++-
 tests/vm/centos           | 82 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 92 insertions(+), 44 deletions(-)
 create mode 100755 tests/vm/centos

-- 
2.14.3

[Qemu-devel] [PATCH 1/3] archive-source.sh: Drop submodule code

[Fam Zheng]

./configure and make now take care of submodules, we only need to clone
the top tree.

Signed-off-by: Fam Zheng <famz@redhat.com>
---
 scripts/archive-source.sh | 47 ++++-------------------------------------------
 1 file changed, 4 insertions(+), 43 deletions(-)

diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index 4e63774f9a..d53774d507 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -18,51 +18,12 @@ if test $# -lt 1; then
     error "Usage: $0 <output tarball>"
 fi
 
-tar_file=`realpath "$1"`
-list_file="${tar_file}.list"
-vroot_dir="${tar_file}.vroot"
+tar_file="$1"
+list_file="$1.list"
 
-# We want a predictable list of submodules for builds, that is
-# independent of what the developer currently has initialized
-# in their checkout, because the build environment is completely
-# different to the host OS.
-submodules="dtc ui/keycodemapdb"
+trap "status=$?; rm -f \"$list_file\"; exit \$status" 0 1 2 3 15
 
-trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 3 15
-
-if git diff-index --quiet HEAD -- &>/dev/null
-then
-    HEAD=HEAD
-else
-    HEAD=`git stash create`
-fi
-git clone --shared . "$vroot_dir"
-test $? -ne 0 && error "failed to clone into '$vroot_dir'"
-
-cd "$vroot_dir"
-test $? -ne 0 && error "failed to change into '$vroot_dir'"
-
-git checkout $HEAD
-test $? -ne 0 && error "failed to checkout $HEAD revision"
-
-for sm in $submodules; do
-    git submodule update --init $sm
-    test $? -ne 0 && error "failed to init submodule $sm"
-done
-
-if test -n "$submodules"; then
-    {
-        git ls-files || error "git ls-files failed"
-        for sm in $submodules; do
-            (cd $sm; git ls-files) | sed "s:^:$sm/:"
-            if test "${PIPESTATUS[*]}" != "0 0"; then
-                error "git ls-files in submodule $sm failed"
-            fi
-        done
-    } | grep -x -v $(for sm in $submodules; do echo "-e $sm"; done) > "$list_file"
-else
-    git ls-files > "$list_file"
-fi
+( git ls-files && echo '.git' ) > "$list_file"
 
 if test $? -ne 0; then
     error "failed to generate list file"
-- 
2.14.3

Re: [Qemu-devel] [PATCH 1/3] archive-source.sh: Drop submodule code

[Daniel P. Berrangé]

On Wed, Apr 04, 2018 at 10:51:59AM +0800, Fam Zheng wrote:
> ./configure and make now take care of submodules, we only need to clone
> the top tree.

If you don't include the submodules in the tar.gz archive, then the
test system needs to be able to clone them from qemu.org, which
requires networking.

The VM tests have networking by default, but the docker tests do
not have networking - you need to opt-in by setting NETWORK=1
make variable. So AFAICT, this will break docker tests.

> 
> Signed-off-by: Fam Zheng <famz@redhat.com>
> ---
>  scripts/archive-source.sh | 47 ++++-------------------------------------------
>  1 file changed, 4 insertions(+), 43 deletions(-)
> 
> diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
> index 4e63774f9a..d53774d507 100755
> --- a/scripts/archive-source.sh
> +++ b/scripts/archive-source.sh
> @@ -18,51 +18,12 @@ if test $# -lt 1; then
>      error "Usage: $0 <output tarball>"
>  fi
>  
> -tar_file=`realpath "$1"`
> -list_file="${tar_file}.list"
> -vroot_dir="${tar_file}.vroot"
> +tar_file="$1"
> +list_file="$1.list"
>  
> -# We want a predictable list of submodules for builds, that is
> -# independent of what the developer currently has initialized
> -# in their checkout, because the build environment is completely
> -# different to the host OS.
> -submodules="dtc ui/keycodemapdb"
> +trap "status=$?; rm -f \"$list_file\"; exit \$status" 0 1 2 3 15
>  
> -trap "status=$?; rm -rf \"$list_file\" \"$vroot_dir\"; exit \$status" 0 1 2 3 15
> -
> -if git diff-index --quiet HEAD -- &>/dev/null
> -then
> -    HEAD=HEAD
> -else
> -    HEAD=`git stash create`
> -fi
> -git clone --shared . "$vroot_dir"
> -test $? -ne 0 && error "failed to clone into '$vroot_dir'"
> -
> -cd "$vroot_dir"
> -test $? -ne 0 && error "failed to change into '$vroot_dir'"
> -
> -git checkout $HEAD
> -test $? -ne 0 && error "failed to checkout $HEAD revision"
> -
> -for sm in $submodules; do
> -    git submodule update --init $sm
> -    test $? -ne 0 && error "failed to init submodule $sm"
> -done
> -
> -if test -n "$submodules"; then
> -    {
> -        git ls-files || error "git ls-files failed"
> -        for sm in $submodules; do
> -            (cd $sm; git ls-files) | sed "s:^:$sm/:"
> -            if test "${PIPESTATUS[*]}" != "0 0"; then
> -                error "git ls-files in submodule $sm failed"
> -            fi
> -        done
> -    } | grep -x -v $(for sm in $submodules; do echo "-e $sm"; done) > "$list_file"
> -else
> -    git ls-files > "$list_file"
> -fi
> +( git ls-files && echo '.git' ) > "$list_file"
>  
>  if test $? -ne 0; then
>      error "failed to generate list file"
> -- 
> 2.14.3
> 
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PATCH 1/3] archive-source.sh: Drop submodule code

[Fam Zheng]

On Wed, 04/04 09:23, Daniel P. Berrangé wrote:
> On Wed, Apr 04, 2018 at 10:51:59AM +0800, Fam Zheng wrote:
> > ./configure and make now take care of submodules, we only need to clone
> > the top tree.
> 
> If you don't include the submodules in the tar.gz archive, then the
> test system needs to be able to clone them from qemu.org, which
> requires networking.
> 
> The VM tests have networking by default, but the docker tests do
> not have networking - you need to opt-in by setting NETWORK=1
> make variable. So AFAICT, this will break docker tests.

Yes, I'll revise the patch.

Fam

Re: [Qemu-devel] [PATCH 1/3] archive-source.sh: Drop submodule code

[Daniel P. Berrangé]

On Wed, Apr 04, 2018 at 05:07:56PM +0800, Fam Zheng wrote:
> On Wed, 04/04 09:23, Daniel P. Berrangé wrote:
> > On Wed, Apr 04, 2018 at 10:51:59AM +0800, Fam Zheng wrote:
> > > ./configure and make now take care of submodules, we only need to clone
> > > the top tree.
> > 
> > If you don't include the submodules in the tar.gz archive, then the
> > test system needs to be able to clone them from qemu.org, which
> > requires networking.
> > 
> > The VM tests have networking by default, but the docker tests do
> > not have networking - you need to opt-in by setting NETWORK=1
> > make variable. So AFAICT, this will break docker tests.
> 
> Yes, I'll revise the patch.

BTW, I can't help feeling that scripts/archive-source.sh should not even
exist. We already have scripts/make-release that has significant overlap
of functionality and is used by 'make dist'. Ideally imho we should be
able to either invoke 'make dist' for testing, or call make-release.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PATCH 1/3] archive-source.sh: Drop submodule code

[Fam Zheng]

On Wed, 04/04 10:24, Daniel P. Berrangé wrote:
> On Wed, Apr 04, 2018 at 05:07:56PM +0800, Fam Zheng wrote:
> > On Wed, 04/04 09:23, Daniel P. Berrangé wrote:
> > > On Wed, Apr 04, 2018 at 10:51:59AM +0800, Fam Zheng wrote:
> > > > ./configure and make now take care of submodules, we only need to clone
> > > > the top tree.
> > > 
> > > If you don't include the submodules in the tar.gz archive, then the
> > > test system needs to be able to clone them from qemu.org, which
> > > requires networking.
> > > 
> > > The VM tests have networking by default, but the docker tests do
> > > not have networking - you need to opt-in by setting NETWORK=1
> > > make variable. So AFAICT, this will break docker tests.
> > 
> > Yes, I'll revise the patch.
> 
> BTW, I can't help feeling that scripts/archive-source.sh should not even
> exist. We already have scripts/make-release that has significant overlap
> of functionality and is used by 'make dist'. Ideally imho we should be
> able to either invoke 'make dist' for testing, or call make-release.

But it has the advantage that it, as test code, never messes with the process to
release QEMU.  :)

Fam

[Qemu-devel] [PATCH 2/3] tests: Add an option for snapshot (default: off)

[Fam Zheng]

Not using snapshot has the benefit of automatically persisting useful
test harnesses, such as docker images and ccache database. Although it
will lose some cleanness, it is imaginably useful for patchew.

Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/vm/basevm.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 686d88decf..960e13f349 100755
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -218,6 +218,8 @@ def parse_args(vm_name):
                       help="build QEMU from source in guest")
     parser.add_option("--interactive", "-I", action="store_true",
                       help="Interactively run command")
+    parser.add_option("--snapshot", "-s", action="store_true",
+                      help="run tests with a snapshot")
     parser.disable_interspersed_args()
     return parser.parse_args()
 
@@ -243,7 +245,10 @@ def main(vmcls):
                    jobs=args.jobs)]
         else:
             cmd = argv
-        vm.boot(args.image + ",snapshot=on")
+        img = args.image
+        if args.snapshot:
+            img += ",snapshot=on"
+        vm.boot(img)
         vm.wait_ssh()
     except Exception as e:
         if isinstance(e, SystemExit) and e.code == 0:
-- 
2.14.3

[Qemu-devel] [PATCH 3/3] tests: Add centos VM testing

[Fam Zheng]

This one does docker testing in the VM. It is intended to replace the
native docker testing on patchew testers.

Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/vm/centos | 82 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)
 create mode 100755 tests/vm/centos

diff --git a/tests/vm/centos b/tests/vm/centos
new file mode 100755
index 0000000000..1a0d1fc0a2
--- /dev/null
+++ b/tests/vm/centos
@@ -0,0 +1,82 @@
+#!/usr/bin/env python
+#
+# CentOS image
+#
+# Copyright 2018 Red Hat Inc.
+#
+# Authors:
+#  Fam Zheng <famz@redhat.com>
+#
+# This code is licensed under the GPL version 2 or later.  See
+# the COPYING file in the top-level directory.
+#
+
+import os
+import sys
+import subprocess
+import basevm
+import time
+
+class CentosVM(basevm.BaseVM):
+    name = "centos"
+    BUILD_SCRIPT = """
+        set -e;
+        cd $(mktemp -d);
+        sudo chmod a+r /dev/vdb;
+        tar -xf /dev/vdb;
+        make docker-test-mingw@fedora V=1 J=8;
+        make docker-test-build@min-glib V=1 J=8;
+    """
+
+    def _gen_cloud_init_iso(self):
+        cidir = self._tmpdir
+        mdata = open(os.path.join(cidir, "meta-data"), "w")
+        mdata.writelines(["instance-id: centos-vm-0\n",
+                          "local-hostname: centos-guest\n"])
+        mdata.close()
+        udata = open(os.path.join(cidir, "user-data"), "w")
+        udata.writelines(["#cloud-config\n",
+                          "chpasswd:\n",
+                          "  list: |\n",
+                          "    root:%s\n" % self.ROOT_PASS,
+                          "    %s:%s\n" % (self.GUEST_USER, self.GUEST_PASS),
+                          "  expire: False\n",
+                          "users:\n",
+                          "  - name: %s\n" % self.GUEST_USER,
+                          "    sudo: ALL=(ALL) NOPASSWD:ALL\n",
+                          "    ssh-authorized-keys:\n",
+                          "    - %s\n" % basevm.SSH_PUB_KEY,
+                          "  - name: root\n",
+                          "    ssh-authorized-keys:\n",
+                          "    - %s\n" % basevm.SSH_PUB_KEY,
+                          "locale: en_US.UTF-8\n"])
+        udata.close()
+        subprocess.check_call(["genisoimage", "-output", "cloud-init.iso",
+                               "-volid", "cidata", "-joliet", "-rock",
+                               "user-data", "meta-data"],
+                               cwd=cidir,
+                               stdin=self._devnull, stdout=self._stdout,
+                               stderr=self._stdout)
+        return os.path.join(cidir, "cloud-init.iso")
+
+    def build_image(self, img):
+        cimg = self._download_with_cache("https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1802.qcow2.xz")
+        img_tmp = img + ".tmp"
+        subprocess.check_call(["cp", "-f", cimg, img_tmp + ".xz"])
+        subprocess.check_call(["xz", "-df", img_tmp + ".xz"])
+        subprocess.check_call(["qemu-img", "resize", img_tmp, "50G"])
+        self.boot(img_tmp, extra_args = ["-cdrom", self._gen_cloud_init_iso()])
+        self.wait_ssh()
+        self.ssh_root_check("touch /etc/cloud/cloud-init.disabled")
+        self.ssh_root_check("yum update -y")
+        self.ssh_root_check("yum install -y docker make git")
+        self.ssh_root_check("systemctl enable docker")
+        self.ssh_root("poweroff")
+        self.wait()
+        if os.path.exists(img):
+            os.remove(img)
+        os.rename(img_tmp, img)
+        return 0
+
+if __name__ == "__main__":
+    sys.exit(basevm.main(CentosVM))
-- 
2.14.3

Re: [Qemu-devel] [PATCH 0/3] Add a CentOS test image to run docker tests

[no-reply]

Hi,

This series failed docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 20180404025201.11921-1-famz@redhat.com
Subject: [Qemu-devel] [PATCH 0/3] Add a CentOS test image to run docker tests

=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
export J=8
time make docker-test-mingw@fedora
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
99d3cb1498 tests: Add centos VM testing
a2d747b22b tests: Add an option for snapshot (default: off)
9410826528 archive-source.sh: Drop submodule code

=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-d015hcwx/src/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
  BUILD   fedora
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-d015hcwx/src'
  GEN     /var/tmp/patchew-tester-tmp-d015hcwx/src/docker-src.2018-04-03-22.58.48.13379/qemu.tar
  COPY    RUNNER
    RUN test-mingw in qemu:fedora 
Packages installed:
PyYAML-3.12-5.fc27.x86_64
SDL-devel-1.2.15-29.fc27.x86_64
bc-1.07.1-3.fc27.x86_64
bison-3.0.4-8.fc27.x86_64
bzip2-1.0.6-24.fc27.x86_64
ccache-3.3.6-1.fc27.x86_64
clang-5.0.1-3.fc27.x86_64
findutils-4.6.0-16.fc27.x86_64
flex-2.6.1-5.fc27.x86_64
gcc-7.3.1-5.fc27.x86_64
gcc-c++-7.3.1-5.fc27.x86_64
gettext-0.19.8.1-12.fc27.x86_64
git-2.14.3-3.fc27.x86_64
glib2-devel-2.54.3-2.fc27.x86_64
hostname-3.18-4.fc27.x86_64
libaio-devel-0.3.110-9.fc27.x86_64
libasan-7.3.1-5.fc27.x86_64
libfdt-devel-1.4.6-1.fc27.x86_64
libubsan-7.3.1-5.fc27.x86_64
llvm-5.0.1-3.fc27.x86_64
make-4.2.1-4.fc27.x86_64
mingw32-SDL-1.2.15-9.fc27.noarch
mingw32-bzip2-1.0.6-9.fc27.noarch
mingw32-curl-7.54.1-2.fc27.noarch
mingw32-glib2-2.54.1-1.fc27.noarch
mingw32-gmp-6.1.2-2.fc27.noarch
mingw32-gnutls-3.5.13-2.fc27.noarch
mingw32-gtk2-2.24.31-4.fc27.noarch
mingw32-gtk3-3.22.16-1.fc27.noarch
mingw32-libjpeg-turbo-1.5.1-3.fc27.noarch
mingw32-libpng-1.6.29-2.fc27.noarch
mingw32-libssh2-1.8.0-3.fc27.noarch
mingw32-libtasn1-4.13-1.fc27.noarch
mingw32-nettle-3.3-3.fc27.noarch
mingw32-pixman-0.34.0-3.fc27.noarch
mingw32-pkg-config-0.28-9.fc27.x86_64
mingw64-SDL-1.2.15-9.fc27.noarch
mingw64-bzip2-1.0.6-9.fc27.noarch
mingw64-curl-7.54.1-2.fc27.noarch
mingw64-glib2-2.54.1-1.fc27.noarch
mingw64-gmp-6.1.2-2.fc27.noarch
mingw64-gnutls-3.5.13-2.fc27.noarch
mingw64-gtk2-2.24.31-4.fc27.noarch
mingw64-gtk3-3.22.16-1.fc27.noarch
mingw64-libjpeg-turbo-1.5.1-3.fc27.noarch
mingw64-libpng-1.6.29-2.fc27.noarch
mingw64-libssh2-1.8.0-3.fc27.noarch
mingw64-libtasn1-4.13-1.fc27.noarch
mingw64-nettle-3.3-3.fc27.noarch
mingw64-pixman-0.34.0-3.fc27.noarch
mingw64-pkg-config-0.28-9.fc27.x86_64
nettle-devel-3.4-1.fc27.x86_64
perl-5.26.1-403.fc27.x86_64
pixman-devel-0.34.0-4.fc27.x86_64
python3-3.6.2-13.fc27.x86_64
sparse-0.5.1-2.fc27.x86_64
tar-1.29-7.fc27.x86_64
which-2.21-4.fc27.x86_64
zlib-devel-1.2.11-4.fc27.x86_64

Environment variables:
TARGET_LIST=
PACKAGES=ccache gettext git tar PyYAML sparse flex bison python3 bzip2 hostname     glib2-devel pixman-devel zlib-devel SDL-devel libfdt-devel     gcc gcc-c++ llvm clang make perl which bc findutils libaio-devel     nettle-devel libasan libubsan     mingw32-pixman mingw32-glib2 mingw32-gmp mingw32-SDL mingw32-pkg-config     mingw32-gtk2 mingw32-gtk3 mingw32-gnutls mingw32-nettle mingw32-libtasn1     mingw32-libjpeg-turbo mingw32-libpng mingw32-curl mingw32-libssh2     mingw32-bzip2     mingw64-pixman mingw64-glib2 mingw64-gmp mingw64-SDL mingw64-pkg-config     mingw64-gtk2 mingw64-gtk3 mingw64-gnutls mingw64-nettle mingw64-libtasn1     mingw64-libjpeg-turbo mingw64-libpng mingw64-curl mingw64-libssh2     mingw64-bzip2
J=8
V=
HOSTNAME=79564714ea88
DEBUG=
SHOW_ENV=1
PWD=/
HOME=/root
CCACHE_DIR=/var/tmp/ccache
DISTTAG=f27container
QEMU_CONFIGURE_OPTS=--python=/usr/bin/python3
FGC=f27
TEST_DIR=/tmp/qemu-test
SHLVL=1
FEATURES=mingw clang pyyaml asan dtc
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAKEFLAGS= -j8
EXTRA_CONFIGURE_OPTS=
_=/usr/bin/env

Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/install --python=/usr/bin/python3 --cross-prefix=x86_64-w64-mingw32- --enable-trace-backends=simple --enable-gnutls --enable-nettle --enable-curl --enable-vnc --enable-bzip2 --enable-guest-agent --with-sdlabi=1.2 --with-gtkabi=2.0
Install prefix    /tmp/qemu-test/install
BIOS directory    /tmp/qemu-test/install
firmware path     /tmp/qemu-test/install/share/qemu-firmware
binary directory  /tmp/qemu-test/install
library directory /tmp/qemu-test/install/lib
module directory  /tmp/qemu-test/install/lib
libexec directory /tmp/qemu-test/install/libexec
include directory /tmp/qemu-test/install/include
config directory  /tmp/qemu-test/install
local state directory   queried at runtime
Windows SDK       no
Source path       /tmp/qemu-test/src
GIT binary        git
GIT submodules    ui/keycodemapdb dtc capstone
C compiler        x86_64-w64-mingw32-gcc
Host C compiler   cc
C++ compiler      x86_64-w64-mingw32-g++
Objective-C compiler clang
ARFLAGS           rv
CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS       -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/pixman-1  -I$(SRC_PATH)/dtc/libfdt -Werror -DHAS_LIBSSH2_SFTP_FSYNC -mms-bitfields -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/glib-2.0 -I/usr/x86_64-w64-mingw32/sys-root/mingw/lib/glib-2.0/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include  -m64 -mcx16 -mthreads -D__USE_MINGW_ANSI_STDIO=1 -DWIN32_LEAN_AND_MEAN -DWINVER=0x501 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include  -I/usr/x86_64-w64-mingw32/sys-root/mingw/include   -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16  -I$(SRC_PATH)/capstone/include
LDFLAGS           -Wl,--nxcompat -Wl,--no-seh -Wl,--dynamicbase -Wl,--warn-common -m64 -g 
make              make
install           install
python            /usr/bin/python3 -B
smbd              /usr/sbin/smbd
module support    no
host CPU          x86_64
host big endian   no
target list       x86_64-softmmu aarch64-softmmu
gprof enabled     no
sparse enabled    no
strip binaries    yes
profiler          no
static build      no
SDL support       yes (1.2.15)
GTK support       yes (2.24.31)
GTK GL support    no
VTE support       no 
TLS priority      NORMAL
GNUTLS support    yes
GNUTLS rnd        yes
libgcrypt         no
libgcrypt kdf     no
nettle            yes (3.3)
nettle kdf        yes
libtasn1          yes
curses support    no
virgl support     no
curl support      yes
mingw32 support   yes
Audio drivers     dsound
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    no
Multipath support no
VNC support       yes
VNC SASL support  no
VNC JPEG support  yes
VNC PNG support   yes
xen support       no
brlapi support    no
bluez  support    no
Documentation     no
PIE               no
vde support       no
netmap support    no
Linux AIO support no
ATTR/XATTR support no
Install blobs     yes
KVM support       no
HAX support       yes
HVF support       no
WHPX support      no
TCG support       yes
TCG debug enabled no
TCG interpreter   no
malloc trim support no
RDMA support      no
fdt support       yes
membarrier        no
preadv support    no
fdatasync         no
madvise           no
posix_madvise     no
posix_memalign    no
libcap-ng support no
vhost-net support no
vhost-crypto support no
vhost-scsi support no
vhost-vsock support no
vhost-user support no
Trace backends    simple
Trace output file trace-<pid>
spice support     no 
rbd support       no
xfsctl support    no
smartcard support no
libusb            no
usb net redir     no
OpenGL support    no
OpenGL dmabufs    no
libiscsi support  no
libnfs support    no
build guest agent yes
QGA VSS support   no
QGA w32 disk info yes
QGA MSI support   no
seccomp support   no
coroutine backend win32
coroutine pool    yes
debug stack usage no
crypto afalg      no
GlusterFS support no
gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   yes
TPM passthrough   no
TPM emulator      no
QOM debugging     yes
Live block migration yes
lzo support       no
snappy support    no
bzip2 support     yes
NUMA host support no
libxml2           no
tcmalloc support  no
jemalloc support  no
avx2 optimization yes
replication support yes
VxHS block device no
capstone          git

WARNING: Use of GTK 2.0 is deprecated and will be removed in
WARNING: future releases. Please switch to using GTK 3.0

WARNING: Use of SDL 1.2 is deprecated and will be removed in
WARNING: future releases. Please switch to using SDL 2.0
  GEN     x86_64-softmmu/config-devices.mak.tmp
  GEN     aarch64-softmmu/config-devices.mak.tmp
  GEN     config-host.h
  GIT     ui/keycodemapdb dtc capstone
  GEN     qemu-options.def
  GEN     qapi-gen
  GEN     trace/generated-tcg-tracers.h
  GEN     trace/generated-helpers-wrappers.h
  GEN     x86_64-softmmu/config-devices.mak
  GEN     aarch64-softmmu/config-devices.mak
  GEN     trace/generated-helpers.h
  GEN     trace/generated-helpers.c
  GEN     module_block.h
  GEN     tests/test-qapi-gen
  GEN     trace-root.h
  GEN     util/trace.h
  GEN     crypto/trace.h
  GEN     io/trace.h
  GEN     migration/trace.h
Submodule 'capstone' (git://git.qemu.org/capstone.git) registered for path 'capstone'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered for path 'ui/keycodemapdb'
Cloning into '/tmp/qemu-test/src/capstone'...
  GEN     block/trace.h
  GEN     chardev/trace.h
  GEN     hw/block/trace.h
  GEN     hw/block/dataplane/trace.h
  GEN     hw/char/trace.h
  GEN     hw/intc/trace.h
  GEN     hw/net/trace.h
fatal: Unable to look up git.qemu.org (port 9418) (Name or service not known)
  GEN     hw/rdma/trace.h
  GEN     hw/rdma/vmw/trace.h
  GEN     hw/virtio/trace.h
fatal: clone of 'git://git.qemu.org/capstone.git' into submodule path '/tmp/qemu-test/src/capstone' failed
Failed to clone 'capstone'. Retry scheduled
Cloning into '/tmp/qemu-test/src/ui/keycodemapdb'...
fatal: Unable to look up git.qemu.org (port 9418) (Name or service not known)
  GEN     hw/audio/trace.h
  GEN     hw/misc/trace.h
  GEN     hw/misc/macio/trace.h
  GEN     hw/usb/trace.h
  GEN     hw/scsi/trace.h
fatal: clone of 'git://git.qemu.org/keycodemapdb.git' into submodule path '/tmp/qemu-test/src/ui/keycodemapdb' failed
Failed to clone 'ui/keycodemapdb'. Retry scheduled
Cloning into '/tmp/qemu-test/src/capstone'...
fatal: Unable to look up git.qemu.org (port 9418) (Name or service not known)
  GEN     hw/nvram/trace.h
  GEN     hw/display/trace.h
  GEN     hw/input/trace.h
fatal: clone of 'git://git.qemu.org/capstone.git' into submodule path '/tmp/qemu-test/src/capstone' failed
Failed to clone 'capstone' a second time, aborting
  GEN     hw/timer/trace.h
./scripts/git-submodule.sh: failed to update modules

Unable to automatically checkout GIT submodules ' ui/keycodemapdb dtc capstone'.
If you require use of an alternative GIT binary (for example to
enable use of a transparent proxy), then please specify it by
running configure by with the '--with-git' argument. e.g.

 $ ./configure --with-git='tsocks git'

Alternatively you may disable automatic GIT submodule checkout
with:

 $ ./configure --disable-git-update

and then manually update submodules prior to running make, with:

 $ scripts/git-submodule.sh update  ui/keycodemapdb dtc capstone

make: *** [Makefile:47: git-submodule-update] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 407, in <module>
    sys.exit(main())
  File "./tests/docker/docker.py", line 404, in main
    return args.cmdobj.run(args, argv)
  File "./tests/docker/docker.py", line 261, in run
    return Docker().run(argv, args.keep, quiet=args.quiet)
  File "./tests/docker/docker.py", line 229, in run
    quiet=quiet)
  File "./tests/docker/docker.py", line 147, in _do_check
    return subprocess.check_call(self._command + cmd, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'run', '--label', 'com.qemu.instance.uuid=3c2ec30637b411e89cb952540069c830', '-u', '0', '--security-opt', 'seccomp=unconfined', '--rm', '--net=none', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=8', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/root/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-d015hcwx/src/docker-src.2018-04-03-22.58.48.13379:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2
make[1]: *** [tests/docker/Makefile.include:129: docker-run] Error 1
make[1]: Leaving directory '/var/tmp/patchew-tester-tmp-d015hcwx/src'
make: *** [tests/docker/Makefile.include:163: docker-run-test-mingw@fedora] Error 2

real	2m41.698s
user	0m1.618s
sys	0m3.855s
=== OUTPUT END ===

Test command exited with code: 2


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [PATCH 0/3] Add a CentOS test image to run docker tests

[no-reply]

Hi,

This series failed docker-build@min-glib build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 20180404025201.11921-1-famz@redhat.com
Subject: [Qemu-devel] [PATCH 0/3] Add a CentOS test image to run docker tests

=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
export J=8
time make docker-test-build@min-glib
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
99d3cb1498 tests: Add centos VM testing
a2d747b22b tests: Add an option for snapshot (default: off)
9410826528 archive-source.sh: Drop submodule code

=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-_g2ozc9x/src/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
  BUILD   min-glib
make[1]: Entering directory '/var/tmp/patchew-tester-tmp-_g2ozc9x/src'
  GEN     /var/tmp/patchew-tester-tmp-_g2ozc9x/src/docker-src.2018-04-03-23.02.08.16350/qemu.tar
  COPY    RUNNER
    RUN test-build in qemu:min-glib 
Environment variables:
HOSTNAME=c7583a32ef00
MAKEFLAGS= -j8
J=8
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
TARGET_LIST=
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
FEATURES= dtc
DEBUG=
_=/usr/bin/env

Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/install
No C++ compiler available; disabling C++ specific optional code
Install prefix    /tmp/qemu-test/install
BIOS directory    /tmp/qemu-test/install/share/qemu
firmware path     /tmp/qemu-test/install/share/qemu-firmware
binary directory  /tmp/qemu-test/install/bin
library directory /tmp/qemu-test/install/lib
module directory  /tmp/qemu-test/install/lib/qemu
libexec directory /tmp/qemu-test/install/libexec
include directory /tmp/qemu-test/install/include
config directory  /tmp/qemu-test/install/etc
local state directory   /tmp/qemu-test/install/var
Manual directory  /tmp/qemu-test/install/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path       /tmp/qemu-test/src
GIT binary        git
GIT submodules    ui/keycodemapdb dtc capstone
C compiler        cc
Host C compiler   cc
C++ compiler      
Objective-C compiler cc
ARFLAGS           rv
CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS       -I/usr/include/pixman-1   -I$(SRC_PATH)/dtc/libfdt -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -fPIE -DPIE -m64 -mcx16 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wendif-labels -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all -Wno-missing-braces -I$(SRC_PATH)/capstone/include
LDFLAGS           -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g 
make              make
install           install
python            python -B
smbd              /usr/sbin/smbd
module support    no
host CPU          x86_64
host big endian   no
target list       x86_64-softmmu aarch64-softmmu
gprof enabled     no
sparse enabled    no
strip binaries    yes
profiler          no
static build      no
SDL support       yes (1.2.14)
GTK support       no 
GTK GL support    no
VTE support       no 
TLS priority      NORMAL
GNUTLS support    no
GNUTLS rnd        no
libgcrypt         no
libgcrypt kdf     no
nettle            no 
nettle kdf        no
libtasn1          no
curses support    no
virgl support     no
curl support      no
mingw32 support   no
Audio drivers     oss
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    no
Multipath support no
VNC support       yes
VNC SASL support  no
VNC JPEG support  no
VNC PNG support   no
xen support       no
brlapi support    no
bluez  support    no
Documentation     no
PIE               yes
vde support       no
netmap support    no
Linux AIO support no
ATTR/XATTR support yes
Install blobs     yes
KVM support       yes
HAX support       no
HVF support       no
WHPX support      no
TCG support       yes
TCG debug enabled no
TCG interpreter   no
malloc trim support yes
RDMA support      no
fdt support       yes
membarrier        no
preadv support    yes
fdatasync         yes
madvise           yes
posix_madvise     yes
posix_memalign    yes
libcap-ng support no
vhost-net support yes
vhost-crypto support yes
vhost-scsi support yes
vhost-vsock support yes
vhost-user support yes
Trace backends    log
spice support     no 
rbd support       no
xfsctl support    no
smartcard support no
libusb            no
usb net redir     no
OpenGL support    no
OpenGL dmabufs    no
libiscsi support  no
libnfs support    no
build guest agent yes
QGA VSS support   no
QGA w32 disk info no
QGA MSI support   no
seccomp support   no
coroutine backend ucontext
coroutine pool    yes
debug stack usage no
crypto afalg      no
GlusterFS support no
gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   no
TPM passthrough   yes
TPM emulator      yes
QOM debugging     yes
Live block migration yes
lzo support       no
snappy support    no
bzip2 support     no
NUMA host support no
libxml2           no
tcmalloc support  no
jemalloc support  no
avx2 optimization no
replication support yes
VxHS block device no
capstone          git

WARNING: Use of SDL 1.2 is deprecated and will be removed in
WARNING: future releases. Please switch to using SDL 2.0
  GEN     x86_64-softmmu/config-devices.mak.tmp
  GEN     aarch64-softmmu/config-devices.mak.tmp
  GEN     config-host.h
  GIT     ui/keycodemapdb dtc capstone
  GEN     qemu-options.def
  GEN     qapi-gen
  GEN     trace/generated-tcg-tracers.h
  GEN     aarch64-softmmu/config-devices.mak
  GEN     x86_64-softmmu/config-devices.mak
  GEN     trace/generated-helpers-wrappers.h
  GEN     trace/generated-helpers.h
  GEN     trace/generated-helpers.c
  GEN     module_block.h
  GEN     tests/test-qapi-gen
  GEN     trace-root.h
  GEN     util/trace.h
  GEN     crypto/trace.h
  GEN     io/trace.h
  GEN     migration/trace.h
  GEN     block/trace.h
  GEN     chardev/trace.h
  GEN     hw/block/trace.h
  GEN     hw/block/dataplane/trace.h
  GEN     hw/char/trace.h
  GEN     hw/intc/trace.h
  GEN     hw/net/trace.h
  GEN     hw/rdma/trace.h
  GEN     hw/rdma/vmw/trace.h
  GEN     hw/virtio/trace.h
  GEN     hw/audio/trace.h
  GEN     hw/misc/trace.h
  GEN     hw/misc/macio/trace.h
  GEN     hw/usb/trace.h
  GEN     hw/scsi/trace.h
  GEN     hw/nvram/trace.h
  GEN     hw/display/trace.h
  GEN     hw/input/trace.h
  GEN     hw/timer/trace.h
  GEN     hw/dma/trace.h
  GEN     hw/sparc/trace.h
  GEN     hw/sparc64/trace.h
  GEN     hw/sd/trace.h
  GEN     hw/isa/trace.h
  GEN     hw/mem/trace.h
  GEN     hw/i386/trace.h
  GEN     hw/i386/xen/trace.h
  GEN     hw/9pfs/trace.h
  GEN     hw/ppc/trace.h
  GEN     hw/pci/trace.h
  GEN     hw/pci-host/trace.h
  GEN     hw/s390x/trace.h
  GEN     hw/vfio/trace.h
  GEN     hw/acpi/trace.h
  GEN     hw/arm/trace.h
  GEN     hw/alpha/trace.h
  GEN     hw/hppa/trace.h
fatal: Unable to look up git.qemu.org (port 9418) (Temporary failure in name resolution)
  GEN     hw/xen/trace.h
  GEN     hw/ide/trace.h
  GEN     hw/tpm/trace.h
Clone of 'git://git.qemu.org/capstone.git' into submodule path 'capstone' failed
./scripts/git-submodule.sh: failed to update modules

Unable to automatically checkout GIT submodules ' ui/keycodemapdb dtc capstone'.
If you require use of an alternative GIT binary (for example to
enable use of a transparent proxy), then please specify it by
running configure by with the '--with-git' argument. e.g.

 $ ./configure --with-git='tsocks git'

Alternatively you may disable automatic GIT submodule checkout
with:

 $ ./configure --disable-git-update

and then manually update submodules prior to running make, with:

 $ scripts/git-submodule.sh update  ui/keycodemapdb dtc capstone

make: *** [git-submodule-update] Error 1
make: *** Waiting for unfinished jobs....
make: *** wait: No child processes.  Stop.
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 407, in <module>
    sys.exit(main())
  File "./tests/docker/docker.py", line 404, in main
    return args.cmdobj.run(args, argv)
  File "./tests/docker/docker.py", line 261, in run
    return Docker().run(argv, args.keep, quiet=args.quiet)
  File "./tests/docker/docker.py", line 229, in run
    quiet=quiet)
  File "./tests/docker/docker.py", line 147, in _do_check
    return subprocess.check_call(self._command + cmd, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'run', '--label', 'com.qemu.instance.uuid=abc5394837b411e8965d52540069c830', '-u', '0', '--security-opt', 'seccomp=unconfined', '--rm', '--net=none', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=8', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/root/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-_g2ozc9x/src/docker-src.2018-04-03-23.02.08.16350:/var/tmp/qemu:z,ro', 'qemu:min-glib', '/var/tmp/qemu/run', 'test-build']' returned non-zero exit status 2
make[1]: *** [tests/docker/Makefile.include:129: docker-run] Error 1
make[1]: Leaving directory '/var/tmp/patchew-tester-tmp-_g2ozc9x/src'
make: *** [tests/docker/Makefile.include:163: docker-run-test-build@min-glib] Error 2

real	2m17.711s
user	0m1.534s
sys	0m3.511s
=== OUTPUT END ===

Test command exited with code: 2


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [PATCH 0/3] Add a CentOS test image to run docker tests

[Peter Xu]

On Wed, Apr 04, 2018 at 10:51:58AM +0800, Fam Zheng wrote:
> Docker testing on patchew has long suffered from 'make check' hangings. The
> cleanness of VM testing is the cure. Now let's add a CentOS 7 image to run the
> tests.  It's purely ad-hoc, but hopefully still easy to understand and use for
> everyone.
> 
> The first patch makes passing source code from host to the container in VM
> working, and is a nice clean up.
> 
> The second patch makes caches work, to speed up repetitive runs like on
> patchew.
> 
> The last patch adds the new image that does the job. Two out of three docker
> tests running on patchew.org are added to the image. I'll wait for Peter to fix
> the 'docker-test-quick@centos6' hanging (oob test) before adding it too.

This hang does not happen on centos7 (tested myself manually).  I
consider this can be related to different default versions of glib2:

centos6: glib2-2.28.8-9.el6.x86_64
centos7: glib2-2.50.3-3.el7.x86_64

(both fetched from docker.io)

qmp-test hangs at oob-test, while QEMU failed to respond to SIGTERM
when trying to take a lock:

#0  0x00007f18eed02334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f18eecfd5d8 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007f18eecfd4a7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f18eb476c9d in ?? () from /lib64/libglib-2.0.so.0
#4  0x0000000000aa6672 in aio_context_unref (ctx=0x225b600) at /root/qemu/util/async.c:497
#5  0x000000000065851c in iothread_instance_finalize (obj=0x225b380) at /root/qemu/iothread.c:129
#6  0x0000000000962d79 in object_deinit (obj=0x225b380, type=0x2199960) at /root/qemu/qom/object.c:462
#7  0x0000000000962e0d in object_finalize (data=0x225b380) at /root/qemu/qom/object.c:476
#8  0x0000000000964146 in object_unref (obj=0x225b380) at /root/qemu/qom/object.c:924
#9  0x0000000000965880 in object_finalize_child_property (obj=0x2257640, name=0x225aca0 "mon_iothread", opaque=0x225b380) at /root/qemu/qom/object.c:1436
#10 0x0000000000962c33 in object_property_del_child (obj=0x2257640, child=0x225b380, errp=0x0) at /root/qemu/qom/object.c:436
#11 0x0000000000962d26 in object_unparent (obj=0x225b380) at /root/qemu/qom/object.c:455
#12 0x0000000000658f00 in iothread_destroy (iothread=0x225b380) at /root/qemu/iothread.c:365
#13 0x00000000004c67a8 in monitor_cleanup () at /root/qemu/monitor.c:4663
#14 0x0000000000669e27 in main (argc=16, argv=0x7ffe33ce3a08, envp=0x7ffe33ce3a90) at /root/qemu/vl.c:4749

Currently I have no quick idea on why the lock was held by anyone
else (I suppose that's the GMainContext's lock), and I'm not sure
whether it could be a glib bug.

-- 
Peter Xu

Re: [Qemu-devel] [PATCH 0/3] Add a CentOS test image to run docker tests

[Peter Xu]

On Wed, Apr 04, 2018 at 01:03:47PM +0800, Peter Xu wrote:
> On Wed, Apr 04, 2018 at 10:51:58AM +0800, Fam Zheng wrote:
> > Docker testing on patchew has long suffered from 'make check' hangings. The
> > cleanness of VM testing is the cure. Now let's add a CentOS 7 image to run the
> > tests.  It's purely ad-hoc, but hopefully still easy to understand and use for
> > everyone.
> > 
> > The first patch makes passing source code from host to the container in VM
> > working, and is a nice clean up.
> > 
> > The second patch makes caches work, to speed up repetitive runs like on
> > patchew.
> > 
> > The last patch adds the new image that does the job. Two out of three docker
> > tests running on patchew.org are added to the image. I'll wait for Peter to fix
> > the 'docker-test-quick@centos6' hanging (oob test) before adding it too.
> 
> This hang does not happen on centos7 (tested myself manually).  I
> consider this can be related to different default versions of glib2:
> 
> centos6: glib2-2.28.8-9.el6.x86_64
> centos7: glib2-2.50.3-3.el7.x86_64
> 
> (both fetched from docker.io)
> 
> qmp-test hangs at oob-test, while QEMU failed to respond to SIGTERM
> when trying to take a lock:
> 
> #0  0x00007f18eed02334 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x00007f18eecfd5d8 in _L_lock_854 () from /lib64/libpthread.so.0
> #2  0x00007f18eecfd4a7 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3  0x00007f18eb476c9d in ?? () from /lib64/libglib-2.0.so.0
> #4  0x0000000000aa6672 in aio_context_unref (ctx=0x225b600) at /root/qemu/util/async.c:497
> #5  0x000000000065851c in iothread_instance_finalize (obj=0x225b380) at /root/qemu/iothread.c:129
> #6  0x0000000000962d79 in object_deinit (obj=0x225b380, type=0x2199960) at /root/qemu/qom/object.c:462
> #7  0x0000000000962e0d in object_finalize (data=0x225b380) at /root/qemu/qom/object.c:476
> #8  0x0000000000964146 in object_unref (obj=0x225b380) at /root/qemu/qom/object.c:924
> #9  0x0000000000965880 in object_finalize_child_property (obj=0x2257640, name=0x225aca0 "mon_iothread", opaque=0x225b380) at /root/qemu/qom/object.c:1436
> #10 0x0000000000962c33 in object_property_del_child (obj=0x2257640, child=0x225b380, errp=0x0) at /root/qemu/qom/object.c:436
> #11 0x0000000000962d26 in object_unparent (obj=0x225b380) at /root/qemu/qom/object.c:455
> #12 0x0000000000658f00 in iothread_destroy (iothread=0x225b380) at /root/qemu/iothread.c:365
> #13 0x00000000004c67a8 in monitor_cleanup () at /root/qemu/monitor.c:4663
> #14 0x0000000000669e27 in main (argc=16, argv=0x7ffe33ce3a08, envp=0x7ffe33ce3a90) at /root/qemu/vl.c:4749
> 
> Currently I have no quick idea on why the lock was held by anyone
> else (I suppose that's the GMainContext's lock), and I'm not sure
> whether it could be a glib bug.

I posted a patch for this to workaround the possible glib bug:

  [PATCH for-2.12] iothread: workaround glib bug which hangs qmp-test

Let's see whether we can accept that.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [PATCH 0/3] Add a CentOS test image to run docker tests

[Daniel P. Berrangé]

On Wed, Apr 04, 2018 at 10:51:58AM +0800, Fam Zheng wrote:
> Docker testing on patchew has long suffered from 'make check' hangings. The
> cleanness of VM testing is the cure. Now let's add a CentOS 7 image to run the
> tests.  It's purely ad-hoc, but hopefully still easy to understand and use for
> everyone.

If our automated tests are hanging, then developers could hit those hangs
to, and depending on where the problem is, end users could hit the hangs
in production deployment.

IOW, hangs in the testing aren't something we should just hide away by
changing the way we run tests - we need to identify & fix the hangs.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PATCH 0/3] Add a CentOS test image to run docker tests

[Fam Zheng]

On Wed, 04/04 09:19, Daniel P. Berrangé wrote:
> On Wed, Apr 04, 2018 at 10:51:58AM +0800, Fam Zheng wrote:
> > Docker testing on patchew has long suffered from 'make check' hangings. The
> > cleanness of VM testing is the cure. Now let's add a CentOS 7 image to run the
> > tests.  It's purely ad-hoc, but hopefully still easy to understand and use for
> > everyone.
> 
> If our automated tests are hanging, then developers could hit those hangs
> to, and depending on where the problem is, end users could hit the hangs
> in production deployment.
> 
> IOW, hangs in the testing aren't something we should just hide away by
> changing the way we run tests - we need to identify & fix the hangs.

This will not hide anything. It just helps how we clean up everything
(especially the hanging test process in patchew) when timeout, and doesn't
affect how we identify and report timeouts.

Fam

Re: [Qemu-devel] [PATCH 0/3] Add a CentOS test image to run docker tests

[Paolo Bonzini]

On 04/04/2018 04:51, Fam Zheng wrote:
> Docker testing on patchew has long suffered from 'make check' hangings. The
> cleanness of VM testing is the cure. Now let's add a CentOS 7 image to run the
> tests.  It's purely ad-hoc, but hopefully still easy to understand and use for
> everyone.
> 
> The first patch makes passing source code from host to the container in VM
> working, and is a nice clean up.
> 
> The second patch makes caches work, to speed up repetitive runs like on
> patchew.
> 
> The last patch adds the new image that does the job. Two out of three docker
> tests running on patchew.org are added to the image. I'll wait for Peter to fix
> the 'docker-test-quick@centos6' hanging (oob test) before adding it too.

I think it's a good idea.  The tests are the same as before, only the
environment is more self-contained.

Paolo

> Fam
> 
> Fam Zheng (3):
>   archive-source.sh: Drop submodule code
>   tests: Add an option for snapshot (default: off)
>   tests: Add centos VM testing
> 
>  scripts/archive-source.sh | 47 +++------------------------
>  tests/vm/basevm.py        |  7 +++-
>  tests/vm/centos           | 82 +++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 92 insertions(+), 44 deletions(-)
>  create mode 100755 tests/vm/centos
>