[PULL 0/6] Improve gitlab-CI and fix a compiler warning

[PULL 0/6] Improve gitlab-CI and fix a compiler warning

[Thomas Huth]

 Hi Peter,

the following changes since commit fd3cd581f9dcd11286daacaa5272e721c65aece8:

  Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20200804' into staging (2020-08-04 18:20:32 +0100)

are available in the Git repository at:

  https://gitlab.com/huth/qemu.git tags/pull-request-2020-08-05

for you to fetch changes up to d2a71d7474d4649eabe554994a3fcba75244cce3:

  Get rid of the libqemustub.a remainders (2020-08-05 11:45:43 +0200)

(I know it's late in the 5.1 cycle, feel free to ignore this if it's
too much for 5.1)

----------------------------------------------------------------
* Test rx-softmmu, avr-softmmu, Centos7 and Debian on gitlab-CI
* Fix compiler warning on 32-bit big endian systems
* Remove remainders of libqemustub.a
----------------------------------------------------------------

Thomas Huth (6):
      tests/docker: Add python3-venv and netcat to the debian-amd64 container
      tests/acceptance: Disable the rx sash and arm cubieboard replay test on Gitlab
      gitlab-ci.yml: Add build-system-debian and build-system-centos jobs
      gitlab-ci: Fix Avocado cache usage
      target/riscv/vector_helper: Fix build on 32-bit big endian hosts
      Get rid of the libqemustub.a remainders

 .gitlab-ci.yml                               | 109 ++++++++++++++++++++++-----
 Makefile                                     |   2 +-
 scripts/coverity-scan/run-coverity-scan      |   3 -
 target/riscv/vector_helper.c                 |   4 +-
 tests/acceptance/machine_rx_gdbsim.py        |   4 +
 tests/acceptance/replay_kernel.py            |   1 +
 tests/docker/dockerfiles/debian-amd64.docker |   4 +-
 tests/test-util-sockets.c                    |   3 +-
 8 files changed, 103 insertions(+), 27 deletions(-)

[PULL 1/6] tests/docker: Add python3-venv and netcat to the debian-amd64 container

[Thomas Huth]

Without python3-venv, I get the following message when trying to
run the acceptance tests within the debian container:

 The virtual environment was not created successfully because ensurepip is not
 available.  On Debian/Ubuntu systems, you need to install the python3-venv
 package using the following command.
    apt-get install python3-venv
 You may need to use sudo with that command.  After installing the python3-venv
 package, recreate your virtual environment.

Let's do it as the message suggests.

And while we're at it, also add netcat here since it is required for
some of the acceptance tests.

Message-Id: <20200730141326.8260-2-thuth@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/docker/dockerfiles/debian-amd64.docker | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
index 8fdfd6a6b0..d2500dcff1 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -20,7 +20,9 @@ RUN apt update && \
         librdmacm-dev \
         libsasl2-dev \
         libsnappy-dev \
-        libvte-dev
+        libvte-dev \
+        netcat-openbsd \
+        python3-venv
 
 # virgl
 RUN apt update && \
-- 
2.18.1

[PULL 2/6] tests/acceptance: Disable the rx sash and arm cubieboard replay test on Gitlab

[Thomas Huth]

These tests always time out on Gitlab, not sure what's happening here.
Let's disable them until somebody has enough spare time to debug the
issues.

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200730141326.8260-3-thuth@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/acceptance/machine_rx_gdbsim.py | 4 ++++
 tests/acceptance/replay_kernel.py     | 1 +
 2 files changed, 5 insertions(+)

diff --git a/tests/acceptance/machine_rx_gdbsim.py b/tests/acceptance/machine_rx_gdbsim.py
index bff63e421d..0c72506028 100644
--- a/tests/acceptance/machine_rx_gdbsim.py
+++ b/tests/acceptance/machine_rx_gdbsim.py
@@ -8,6 +8,9 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import os
+
+from avocado import skipIf
 from avocado_qemu import Test
 from avocado_qemu import exec_command_and_wait_for_pattern
 from avocado_qemu import wait_for_console_pattern
@@ -42,6 +45,7 @@ class RxGdbSimMachine(Test):
         # FIXME limit baudrate on chardev, else we type too fast
         #exec_command_and_wait_for_pattern(self, 'version', gcc_version)
 
+    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
     def test_linux_sash(self):
         """
         Boots a Linux kernel and checks that the console is operational.
diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
index 62d2db8c64..b79fc8daf8 100644
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -126,6 +126,7 @@ class ReplayKernel(LinuxKernelTest):
 
         self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1)
 
+    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
     def test_arm_cubieboard_initrd(self):
         """
         :avocado: tags=arch:arm
-- 
2.18.1

[PULL 3/6] gitlab-ci.yml: Add build-system-debian and build-system-centos jobs

[Thomas Huth]

We were missing the two new targets avr-softmmu and rx-softmmu in the
gitlab-CI so far, and did not add some of the "other endianess" targets
like sh4eb-softmmu yet.
Since the current build-system-* jobs run already for a very long time,
let's do not add these missing targets there, but introduce two new
additional build jobs, one running with Debian and one running with
CentOS, and add the new targets there. Also move some targets from
the old build-system-* jobs to these new targets, to distribute the
load and reduce the runtime of the CI.

Message-Id: <20200730141326.8260-4-thuth@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 88 +++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 73 insertions(+), 15 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 362e5ee755..e96bcd50f8 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -53,68 +53,126 @@ include:
     - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
     - du -chs $HOME/avocado/data/cache
 
-build-system-ubuntu-main:
+build-system-ubuntu:
   <<: *native_build_job_definition
   variables:
     IMAGE: ubuntu2004
-    TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu lm32-softmmu
-      moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu ppc-softmmu
-      riscv64-softmmu sparc-softmmu
+    TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
+      moxie-softmmu microblazeel-softmmu mips64el-softmmu
     MAKE_CHECK_ARGS: check-build
   artifacts:
     paths:
       - build
 
-check-system-ubuntu-main:
+check-system-ubuntu:
   <<: *native_test_job_definition
   needs:
-    - job: build-system-ubuntu-main
+    - job: build-system-ubuntu
       artifacts: true
   variables:
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check
 
-acceptance-system-ubuntu-main:
+acceptance-system-ubuntu:
   <<: *native_test_job_definition
   needs:
-    - job: build-system-ubuntu-main
+    - job: build-system-ubuntu
       artifacts: true
   variables:
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check-acceptance
   <<: *post_acceptance
 
-build-system-fedora-alt:
+build-system-debian:
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: debian-amd64
+    TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
+      riscv64-softmmu sh4eb-softmmu sparc-softmmu xtensaeb-softmmu
+    MAKE_CHECK_ARGS: check-build
+  artifacts:
+    paths:
+      - build
+
+check-system-debian:
+  <<: *native_test_job_definition
+  needs:
+    - job: build-system-debian
+      artifacts: true
+  variables:
+    IMAGE: debian-amd64
+    MAKE_CHECK_ARGS: check
+
+acceptance-system-debian:
+  <<: *native_test_job_definition
+  needs:
+    - job: build-system-debian
+      artifacts: true
+  variables:
+    IMAGE: debian-amd64
+    MAKE_CHECK_ARGS: check-acceptance
+  <<: *post_acceptance
+
+build-system-fedora:
   <<: *native_build_job_definition
   variables:
     IMAGE: fedora
     TARGETS: tricore-softmmu unicore32-softmmu microblaze-softmmu mips-softmmu
-      riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu x86_64-softmmu
-      xtensa-softmmu nios2-softmmu or1k-softmmu
+      xtensa-softmmu m68k-softmmu riscv32-softmmu ppc-softmmu sparc64-softmmu
     MAKE_CHECK_ARGS: check-build
   artifacts:
     paths:
       - build
 
-check-system-fedora-alt:
+check-system-fedora:
   <<: *native_test_job_definition
   needs:
-    - job: build-system-fedora-alt
+    - job: build-system-fedora
       artifacts: true
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check
 
-acceptance-system-fedora-alt:
+acceptance-system-fedora:
   <<: *native_test_job_definition
   needs:
-    - job: build-system-fedora-alt
+    - job: build-system-fedora
       artifacts: true
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
   <<: *post_acceptance
 
+build-system-centos:
+  <<: *native_build_job_definition
+  variables:
+    IMAGE: centos8
+    TARGETS: ppc64-softmmu lm32-softmmu or1k-softmmu s390x-softmmu
+      x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
+    MAKE_CHECK_ARGS: check-build
+  artifacts:
+    paths:
+      - build
+
+check-system-centos:
+  <<: *native_test_job_definition
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos8
+    MAKE_CHECK_ARGS: check
+
+acceptance-system-centos:
+  <<: *native_test_job_definition
+  needs:
+    - job: build-system-centos
+      artifacts: true
+  variables:
+    IMAGE: centos8
+    MAKE_CHECK_ARGS: check-acceptance
+  <<: *post_acceptance
+
 build-disabled:
   <<: *native_build_job_definition
   variables:
-- 
2.18.1

[PULL 4/6] gitlab-ci: Fix Avocado cache usage

[Thomas Huth]

In commit 6957fd98dc ("gitlab: add avocado asset caching") we
tried to save the Avocado cache (as in commit c1073e44b4 with
Travis-CI) however it doesn't work as expected. For some reason
Avocado uses /root/avocado_cache/ which we can not select later.

Manually generate a Avocado config to force the use of the
current job's directory.

This patch is based on an earlier version from Philippe Mathieu-Daudé.

Message-Id: <20200730141326.8260-5-thuth@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 25 +++++++++++++++++++------
 1 file changed, 19 insertions(+), 6 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index e96bcd50f8..9820066379 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -47,11 +47,24 @@ include:
     - find . -type f -exec touch {} +
     - make $MAKE_CHECK_ARGS
 
-.post_acceptance_template: &post_acceptance
+.acceptance_template: &acceptance_definition
+  cache:
+    key: "${CI_JOB_NAME}-cache"
+    paths:
+      - ${CI_PROJECT_DIR}/avocado-cache
+    policy: pull-push
+  before_script:
+    - mkdir -p ~/.config/avocado
+    - echo "[datadir.paths]" > ~/.config/avocado/avocado.conf
+    - echo "cache_dirs = ['${CI_PROJECT_DIR}/avocado-cache']"
+           >> ~/.config/avocado/avocado.conf
+    - if [ -d ${CI_PROJECT_DIR}/avocado-cache ]; then
+        du -chs ${CI_PROJECT_DIR}/avocado-cache ;
+      fi
   after_script:
     - cd build
     - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
-    - du -chs $HOME/avocado/data/cache
+    - du -chs ${CI_PROJECT_DIR}/avocado-cache
 
 build-system-ubuntu:
   <<: *native_build_job_definition
@@ -81,7 +94,7 @@ acceptance-system-ubuntu:
   variables:
     IMAGE: ubuntu2004
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *post_acceptance
+  <<: *acceptance_definition
 
 build-system-debian:
   <<: *native_build_job_definition
@@ -111,7 +124,7 @@ acceptance-system-debian:
   variables:
     IMAGE: debian-amd64
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *post_acceptance
+  <<: *acceptance_definition
 
 build-system-fedora:
   <<: *native_build_job_definition
@@ -141,7 +154,7 @@ acceptance-system-fedora:
   variables:
     IMAGE: fedora
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *post_acceptance
+  <<: *acceptance_definition
 
 build-system-centos:
   <<: *native_build_job_definition
@@ -171,7 +184,7 @@ acceptance-system-centos:
   variables:
     IMAGE: centos8
     MAKE_CHECK_ARGS: check-acceptance
-  <<: *post_acceptance
+  <<: *acceptance_definition
 
 build-disabled:
   <<: *native_build_job_definition
-- 
2.18.1

[PULL 5/6] target/riscv/vector_helper: Fix build on 32-bit big endian hosts

[Thomas Huth]

The code currently fails to compile on 32-bit big endian hosts:

 target/riscv/vector_helper.c: In function 'vext_clear':
 target/riscv/vector_helper.c:154:16: error: cast to pointer from integer
 of different size [-Werror=int-to-pointer-cast]
         memset((void *)((uintptr_t)tail & ~(7ULL)), 0, part1);
                ^
 target/riscv/vector_helper.c:155:16: error: cast to pointer from integer
 of different size [-Werror=int-to-pointer-cast]
         memset((void *)(((uintptr_t)tail + 8) & ~(7ULL)), 0, part2);
                ^
 cc1: all warnings being treated as errors

We should not use "long long" (i.e. 64-bit) values here to avoid the
problem. Switch to our QEMU_ALIGN_PTR_DOWN/UP macros instead.

Fixes: 751538d5da ("add vector stride load and store instructions")
Suggested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200804170055.2851-3-thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 target/riscv/vector_helper.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/target/riscv/vector_helper.c b/target/riscv/vector_helper.c
index 39f44d1029..793af99067 100644
--- a/target/riscv/vector_helper.c
+++ b/target/riscv/vector_helper.c
@@ -151,8 +151,8 @@ static void vext_clear(void *tail, uint32_t cnt, uint32_t tot)
     if (cnt % 8) {
         part1 = 8 - (cnt % 8);
         part2 = tot - cnt - part1;
-        memset((void *)((uintptr_t)tail & ~(7ULL)), 0, part1);
-        memset((void *)(((uintptr_t)tail + 8) & ~(7ULL)), 0, part2);
+        memset(QEMU_ALIGN_PTR_DOWN(tail, 8), 0, part1);
+        memset(QEMU_ALIGN_PTR_UP(tail, 8), 0, part2);
     } else {
         memset(tail, 0, part2);
     }
-- 
2.18.1

[PULL 6/6] Get rid of the libqemustub.a remainders

[Thomas Huth]

libqemustub.a has been removed in commit ebedb37c8d ("Makefile: Remove
libqemustub.a"). Some remainders have been missed. Remove them now.

Message-Id: <20200804170055.2851-8-thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 Makefile                                | 2 +-
 scripts/coverity-scan/run-coverity-scan | 3 ---
 tests/test-util-sockets.c               | 3 ++-
 3 files changed, 3 insertions(+), 5 deletions(-)

diff --git a/Makefile b/Makefile
index c2120d8d48..13dd708c4a 100644
--- a/Makefile
+++ b/Makefile
@@ -726,7 +726,7 @@ virtiofsd$(EXESUF): $(virtiofsd-obj-y) libvhost-user.a $(COMMON_LDADDS)
 	$(call LINK, $^)
 endif
 
-vhost-user-gpu$(EXESUF): $(vhost-user-gpu-obj-y) $(libvhost-user-obj-y) libqemuutil.a libqemustub.a
+vhost-user-gpu$(EXESUF): $(vhost-user-gpu-obj-y) $(libvhost-user-obj-y) libqemuutil.a
 	$(call LINK, $^)
 
 ifdef CONFIG_VHOST_USER_INPUT
diff --git a/scripts/coverity-scan/run-coverity-scan b/scripts/coverity-scan/run-coverity-scan
index 03a791dec9..6eefb4b558 100755
--- a/scripts/coverity-scan/run-coverity-scan
+++ b/scripts/coverity-scan/run-coverity-scan
@@ -403,9 +403,6 @@ echo "Configuring..."
     --enable-mpath --enable-libxml2 --enable-glusterfs \
     --enable-virtfs --enable-zstd
 
-echo "Making libqemustub.a..."
-make libqemustub.a
-
 echo "Running cov-build..."
 rm -rf cov-int
 mkdir cov-int
diff --git a/tests/test-util-sockets.c b/tests/test-util-sockets.c
index 2ca1e99f17..261dc48c03 100644
--- a/tests/test-util-sockets.c
+++ b/tests/test-util-sockets.c
@@ -64,7 +64,8 @@ int monitor_get_fd(Monitor *mon, const char *fdname, Error **errp)
     return dup(mon_fd);
 }
 
-/* Syms in libqemustub.a are discarded at .o file granularity.
+/*
+ * Syms of stubs in libqemuutil.a are discarded at .o file granularity.
  * To replace monitor_get_fd() we must ensure everything in
  * stubs/monitor.c is defined, to make sure monitor.o is discarded
  * otherwise we get duplicate syms at link time.
-- 
2.18.1

Re: [PULL 0/6] Improve gitlab-CI and fix a compiler warning

[Peter Maydell]

On Wed, 5 Aug 2020 at 10:58, Thomas Huth <thuth@redhat.com> wrote:
>
>  Hi Peter,
>
> the following changes since commit fd3cd581f9dcd11286daacaa5272e721c65aece8:
>
>   Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20200804' into staging (2020-08-04 18:20:32 +0100)
>
> are available in the Git repository at:
>
>   https://gitlab.com/huth/qemu.git tags/pull-request-2020-08-05
>
> for you to fetch changes up to d2a71d7474d4649eabe554994a3fcba75244cce3:
>
>   Get rid of the libqemustub.a remainders (2020-08-05 11:45:43 +0200)
>
> (I know it's late in the 5.1 cycle, feel free to ignore this if it's
> too much for 5.1)
>
> ----------------------------------------------------------------
> * Test rx-softmmu, avr-softmmu, Centos7 and Debian on gitlab-CI
> * Fix compiler warning on 32-bit big endian systems
> * Remove remainders of libqemustub.a
> ----------------------------------------------------------------


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/5.1
for any user-visible changes.

-- PMM

Re: [PULL 0/6] Improve gitlab-CI and fix a compiler warning

[Alistair Francis]

On Wed, Aug 5, 2020 at 5:58 AM Thomas Huth <thuth@redhat.com> wrote:
>
>  Hi Peter,
>
> the following changes since commit fd3cd581f9dcd11286daacaa5272e721c65aece8:
>
>   Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20200804' into staging (2020-08-04 18:20:32 +0100)
>
> are available in the Git repository at:
>
>   https://gitlab.com/huth/qemu.git tags/pull-request-2020-08-05
>
> for you to fetch changes up to d2a71d7474d4649eabe554994a3fcba75244cce3:
>
>   Get rid of the libqemustub.a remainders (2020-08-05 11:45:43 +0200)
>
> (I know it's late in the 5.1 cycle, feel free to ignore this if it's
> too much for 5.1)
>
> ----------------------------------------------------------------
> * Test rx-softmmu, avr-softmmu, Centos7 and Debian on gitlab-CI
> * Fix compiler warning on 32-bit big endian systems
> * Remove remainders of libqemustub.a
> ----------------------------------------------------------------
>
> Thomas Huth (6):
>       tests/docker: Add python3-venv and netcat to the debian-amd64 container
>       tests/acceptance: Disable the rx sash and arm cubieboard replay test on Gitlab
>       gitlab-ci.yml: Add build-system-debian and build-system-centos jobs
>       gitlab-ci: Fix Avocado cache usage
>       target/riscv/vector_helper: Fix build on 32-bit big endian hosts
>       Get rid of the libqemustub.a remainders

Thanks for the RISC-V fix, I'm currently on holidays which is why I
haven't reviewed it.

Alistair

>
>  .gitlab-ci.yml                               | 109 ++++++++++++++++++++++-----
>  Makefile                                     |   2 +-
>  scripts/coverity-scan/run-coverity-scan      |   3 -
>  target/riscv/vector_helper.c                 |   4 +-
>  tests/acceptance/machine_rx_gdbsim.py        |   4 +
>  tests/acceptance/replay_kernel.py            |   1 +
>  tests/docker/dockerfiles/debian-amd64.docker |   4 +-
>  tests/test-util-sockets.c                    |   3 +-
>  8 files changed, 103 insertions(+), 27 deletions(-)
>
>