[PATCH v2 0/4] travis-ci: Add a KVM-only s390x job

[PATCH v2 0/4] travis-ci: Add a KVM-only s390x job

[Philippe Mathieu-Daudé]

Add a Travis job to build a KVM-only QEMU on s390x.

This series also contains few fixes for Travis/s390x.

Since v1:
- Do not disable autoconverge on s390x, but reduce the test
  initial bandwidth (dgilbert)
- Added danpb R-b tags

Philippe Mathieu-Daudé (4):
  tests/test-util-filemonitor: Fix Travis-CI $ARCH env variable name
  tests/test-util-sockets: Skip test on non-x86 Travis containers
  tests/migration: Reduce autoconverge initial bandwidth
  .travis.yml: Add a KVM-only s390x job

 tests/qtest/migration-test.c  |  2 +-
 tests/test-util-filemonitor.c |  2 +-
 tests/test-util-sockets.c     |  7 ++++++
 .travis.yml                   | 42 +++++++++++++++++++++++++++++++++++
 4 files changed, 51 insertions(+), 2 deletions(-)

-- 
2.21.1

[PATCH-for-5.0 v2 1/4] tests/test-util-filemonitor: Fix Travis-CI $ARCH env variable name

[Philippe Mathieu-Daudé]

While we can find reference of a 'TRAVIS_ARCH' variable in
the environment and source [1], per the Travis-CI multi-arch
documentation [2] the variable is named TRAVIS_CPU_ARCH.

[1] https://github.com/travis-ci/travis-build/blob/v10.0.0/lib/travis/build/bash/travis_setup_env.bash#L39
[2] https://docs.travis-ci.com/user/multi-cpu-architectures/#identifying-cpu-architecture-of-build-jobs

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/test-util-filemonitor.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/test-util-filemonitor.c b/tests/test-util-filemonitor.c
index 45009c69f4..e703a7f8fc 100644
--- a/tests/test-util-filemonitor.c
+++ b/tests/test-util-filemonitor.c
@@ -415,7 +415,7 @@ test_file_monitor_events(void)
      * This test does not work on Travis LXD containers since some
      * syscalls are blocked in that environment.
      */
-    travis_arch = getenv("TRAVIS_ARCH");
+    travis_arch = getenv("TRAVIS_CPU_ARCH");
     if (travis_arch && !g_str_equal(travis_arch, "x86_64")) {
         g_test_skip("Test does not work on non-x86 Travis containers.");
         return;
-- 
2.21.1

[PATCH-for-5.0 v2 2/4] tests/test-util-sockets: Skip test on non-x86 Travis containers

[Philippe Mathieu-Daudé]

Similarly to commit 4f370b1098, test-util-sockets fails in
restricted non-x86 Travis containers since they apparently
blacklisted some required system calls there.
Let's simply skip the test if we detect such an environment.

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/test-util-sockets.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/tests/test-util-sockets.c b/tests/test-util-sockets.c
index 5fd947c7bf..046ebec8ba 100644
--- a/tests/test-util-sockets.c
+++ b/tests/test-util-sockets.c
@@ -231,11 +231,18 @@ static void test_socket_fd_pass_num_nocli(void)
 int main(int argc, char **argv)
 {
     bool has_ipv4, has_ipv6;
+    char *travis_arch;
 
     socket_init();
 
     g_test_init(&argc, &argv, NULL);
 
+    travis_arch = getenv("TRAVIS_CPU_ARCH");
+    if (travis_arch && !g_str_equal(travis_arch, "x86_64")) {
+        g_printerr("Test does not work on non-x86 Travis containers.");
+        goto end;
+    }
+
     /* We're creating actual IPv4/6 sockets, so we should
      * check if the host running tests actually supports
      * each protocol to avoid breaking tests on machines
-- 
2.21.1

Re: [PATCH-for-5.0 v2 2/4] tests/test-util-sockets: Skip test on non-x86 Travis containers

[Cornelia Huck]

On Wed, 18 Mar 2020 23:27:15 +0100
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> Similarly to commit 4f370b1098, test-util-sockets fails in
> restricted non-x86 Travis containers since they apparently
> blacklisted some required system calls there.

Is "they" == "Travis admins"? Can we get them to remove those calls
from the blacklist?

(I'm wondering why x86 allows those calls. Probably just because it has
been around for longer.)

> Let's simply skip the test if we detect such an environment.
> 
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/test-util-sockets.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/tests/test-util-sockets.c b/tests/test-util-sockets.c
> index 5fd947c7bf..046ebec8ba 100644
> --- a/tests/test-util-sockets.c
> +++ b/tests/test-util-sockets.c
> @@ -231,11 +231,18 @@ static void test_socket_fd_pass_num_nocli(void)
>  int main(int argc, char **argv)
>  {
>      bool has_ipv4, has_ipv6;
> +    char *travis_arch;
>  
>      socket_init();
>  
>      g_test_init(&argc, &argv, NULL);
>  
> +    travis_arch = getenv("TRAVIS_CPU_ARCH");
> +    if (travis_arch && !g_str_equal(travis_arch, "x86_64")) {
> +        g_printerr("Test does not work on non-x86 Travis containers.");
> +        goto end;
> +    }
> +
>      /* We're creating actual IPv4/6 sockets, so we should
>       * check if the host running tests actually supports
>       * each protocol to avoid breaking tests on machines

Re: [PATCH-for-5.0 v2 2/4] tests/test-util-sockets: Skip test on non-x86 Travis containers

[Philippe Mathieu-Daudé]

On 3/20/20 12:39 PM, Cornelia Huck wrote:
> On Wed, 18 Mar 2020 23:27:15 +0100
> Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> 
>> Similarly to commit 4f370b1098, test-util-sockets fails in
>> restricted non-x86 Travis containers since they apparently
>> blacklisted some required system calls there.
> 
> Is "they" == "Travis admins"? Can we get them to remove those calls
> from the blacklist?

I suppose, I copy/pasted Thomas's description from commit 4f370b1098.

No clue, but we can try :)

> 
> (I'm wondering why x86 allows those calls. Probably just because it has
> been around for longer.)
> 
>> Let's simply skip the test if we detect such an environment.
>>
>> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>   tests/test-util-sockets.c | 7 +++++++
>>   1 file changed, 7 insertions(+)
>>
>> diff --git a/tests/test-util-sockets.c b/tests/test-util-sockets.c
>> index 5fd947c7bf..046ebec8ba 100644
>> --- a/tests/test-util-sockets.c
>> +++ b/tests/test-util-sockets.c
>> @@ -231,11 +231,18 @@ static void test_socket_fd_pass_num_nocli(void)
>>   int main(int argc, char **argv)
>>   {
>>       bool has_ipv4, has_ipv6;
>> +    char *travis_arch;
>>   
>>       socket_init();
>>   
>>       g_test_init(&argc, &argv, NULL);
>>   
>> +    travis_arch = getenv("TRAVIS_CPU_ARCH");
>> +    if (travis_arch && !g_str_equal(travis_arch, "x86_64")) {
>> +        g_printerr("Test does not work on non-x86 Travis containers.");
>> +        goto end;
>> +    }
>> +
>>       /* We're creating actual IPv4/6 sockets, so we should
>>        * check if the host running tests actually supports
>>        * each protocol to avoid breaking tests on machines
> 

Re: [PATCH-for-5.0 v2 2/4] tests/test-util-sockets: Skip test on non-x86 Travis containers

[Daniel P. Berrangé]

On Fri, Mar 20, 2020 at 12:39:58PM +0100, Cornelia Huck wrote:
> On Wed, 18 Mar 2020 23:27:15 +0100
> Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> 
> > Similarly to commit 4f370b1098, test-util-sockets fails in
> > restricted non-x86 Travis containers since they apparently
> > blacklisted some required system calls there.
> 
> Is "they" == "Travis admins"? Can we get them to remove those calls
> from the blacklist?
> 
> (I'm wondering why x86 allows those calls. Probably just because it has
> been around for longer.)

I believe it is due to the different build env architecture for
the non-x86 arches, which is based on a strict locked down container,
compared to x86 which IIUC was a full VM.

Blacklisting like this is a bit hacky, but a prudent fix in the short
term.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH-for-5.0 v2 2/4] tests/test-util-sockets: Skip test on non-x86 Travis containers

[Philippe Mathieu-Daudé]

On 3/23/20 10:48 AM, Daniel P. Berrangé wrote:
> On Fri, Mar 20, 2020 at 12:39:58PM +0100, Cornelia Huck wrote:
>> On Wed, 18 Mar 2020 23:27:15 +0100
>> Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>>
>>> Similarly to commit 4f370b1098, test-util-sockets fails in
>>> restricted non-x86 Travis containers since they apparently
>>> blacklisted some required system calls there.
>>
>> Is "they" == "Travis admins"? Can we get them to remove those calls
>> from the blacklist?
>>
>> (I'm wondering why x86 allows those calls. Probably just because it has
>> been around for longer.)
> 
> I believe it is due to the different build env architecture for
> the non-x86 arches, which is based on a strict locked down container,
> compared to x86 which IIUC was a full VM.

Indeed, Thomas described:

  "Unfortunately, the builds are running in quite restricted
   LXD containers there, for example it is not possible to
   create huge files there (even if they are just sparse),
   and certain system calls are blocked. So we have to change
   some tests first to stop them failing in such environments."

https://www.mail-archive.com/qemu-devel@nongnu.org/msg659909.html

> 
> Blacklisting like this is a bit hacky, but a prudent fix in the short
> term.
> 
> Regards,
> Daniel
> 

[PATCH-for-5.0 v2 3/4] tests/migration: Reduce autoconverge initial bandwidth

[Philippe Mathieu-Daudé]

When using max-bandwidth=~100Mb/s, this test fails on Travis-CI
s390x when configured with --disable-tcg:

  $ make check-qtest
    TEST    check-qtest-s390x: tests/qtest/boot-serial-test
  qemu-system-s390x: -accel tcg: invalid accelerator tcg
  qemu-system-s390x: falling back to KVM
    TEST    check-qtest-s390x: tests/qtest/pxe-test
    TEST    check-qtest-s390x: tests/qtest/test-netfilter
    TEST    check-qtest-s390x: tests/qtest/test-filter-mirror
    TEST    check-qtest-s390x: tests/qtest/test-filter-redirector
    TEST    check-qtest-s390x: tests/qtest/drive_del-test
    TEST    check-qtest-s390x: tests/qtest/device-plug-test
    TEST    check-qtest-s390x: tests/qtest/virtio-ccw-test
    TEST    check-qtest-s390x: tests/qtest/cpu-plug-test
    TEST    check-qtest-s390x: tests/qtest/migration-test
  **
  ERROR:tests/qtest/migration-test.c:1229:test_migrate_auto_converge: 'got_stop' should be FALSE
  ERROR - Bail out! ERROR:tests/qtest/migration-test.c:1229:test_migrate_auto_converge: 'got_stop' should be FALSE
  make: *** [tests/Makefile.include:633: check-qtest-s390x] Error 1

Per David Gilbert, "it could just be the writing is slow on s390
and the migration thread fast; in which case the autocomplete
wouldn't be needed. Perhaps we just need to reduce the bandwidth
limit."

Tuning the threshold by reducing the initial bandwidth makes the
autoconverge test pass.

Suggested-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/qtest/migration-test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
index 3d6cc83b88..727a97cf87 100644
--- a/tests/qtest/migration-test.c
+++ b/tests/qtest/migration-test.c
@@ -1211,7 +1211,7 @@ static void test_migrate_auto_converge(void)
      * without throttling.
      */
     migrate_set_parameter_int(from, "downtime-limit", 1);
-    migrate_set_parameter_int(from, "max-bandwidth", 100000000); /* ~100Mb/s */
+    migrate_set_parameter_int(from, "max-bandwidth", 10000000000); /* ~10Gb/s */
 
     /* To check remaining size after precopy */
     migrate_set_capability(from, "pause-before-switchover", true);
-- 
2.21.1

Re: [PATCH-for-5.0 v2 3/4] tests/migration: Reduce autoconverge initial bandwidth

[Dr. David Alan Gilbert]

* Philippe Mathieu-Daudé (philmd@redhat.com) wrote:
> When using max-bandwidth=~100Mb/s, this test fails on Travis-CI
> s390x when configured with --disable-tcg:
> 
>   $ make check-qtest
>     TEST    check-qtest-s390x: tests/qtest/boot-serial-test
>   qemu-system-s390x: -accel tcg: invalid accelerator tcg
>   qemu-system-s390x: falling back to KVM
>     TEST    check-qtest-s390x: tests/qtest/pxe-test
>     TEST    check-qtest-s390x: tests/qtest/test-netfilter
>     TEST    check-qtest-s390x: tests/qtest/test-filter-mirror
>     TEST    check-qtest-s390x: tests/qtest/test-filter-redirector
>     TEST    check-qtest-s390x: tests/qtest/drive_del-test
>     TEST    check-qtest-s390x: tests/qtest/device-plug-test
>     TEST    check-qtest-s390x: tests/qtest/virtio-ccw-test
>     TEST    check-qtest-s390x: tests/qtest/cpu-plug-test
>     TEST    check-qtest-s390x: tests/qtest/migration-test
>   **
>   ERROR:tests/qtest/migration-test.c:1229:test_migrate_auto_converge: 'got_stop' should be FALSE
>   ERROR - Bail out! ERROR:tests/qtest/migration-test.c:1229:test_migrate_auto_converge: 'got_stop' should be FALSE
>   make: *** [tests/Makefile.include:633: check-qtest-s390x] Error 1
> 
> Per David Gilbert, "it could just be the writing is slow on s390
> and the migration thread fast; in which case the autocomplete
> wouldn't be needed. Perhaps we just need to reduce the bandwidth
> limit."
> 
> Tuning the threshold by reducing the initial bandwidth makes the
> autoconverge test pass.
> 
> Suggested-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  tests/qtest/migration-test.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
> index 3d6cc83b88..727a97cf87 100644
> --- a/tests/qtest/migration-test.c
> +++ b/tests/qtest/migration-test.c
> @@ -1211,7 +1211,7 @@ static void test_migrate_auto_converge(void)
>       * without throttling.
>       */
>      migrate_set_parameter_int(from, "downtime-limit", 1);
> -    migrate_set_parameter_int(from, "max-bandwidth", 100000000); /* ~100Mb/s */
> +    migrate_set_parameter_int(from, "max-bandwidth", 10000000000); /* ~10Gb/s */


Hmm - I was expecting you to need to *reduce* the speed;
now I'm very confused.
You should get 'stop' if the migration completed too soon - surely
incresing the bandwidth would make that *more* likely to happen.

Dave


>      /* To check remaining size after precopy */
>      migrate_set_capability(from, "pause-before-switchover", true);
> -- 
> 2.21.1
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

[PATCH v2 4/4] .travis.yml: Add a KVM-only s390x job

[Philippe Mathieu-Daudé]

Add a job to build QEMU on s390x with TCG disabled, so
this configuration won't bitrot over time.

This job is quick, running check-unit: Ran for 4 min 48 sec
https://travis-ci.org/github/philmd/qemu/jobs/663659486

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index b92798ac3b..0109f02ed4 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -524,6 +524,48 @@ jobs:
               $(exit $BUILD_RC);
           fi
 
+    - name: "[s390x] GCC check (KVM)"
+      arch: s390x
+      dist: bionic
+      addons:
+        apt_packages:
+          - libaio-dev
+          - libattr1-dev
+          - libbrlapi-dev
+          - libcap-ng-dev
+          - libgcrypt20-dev
+          - libgnutls28-dev
+          - libgtk-3-dev
+          - libiscsi-dev
+          - liblttng-ust-dev
+          - libncurses5-dev
+          - libnfs-dev
+          - libnss3-dev
+          - libpixman-1-dev
+          - libpng-dev
+          - librados-dev
+          - libsdl2-dev
+          - libseccomp-dev
+          - liburcu-dev
+          - libusb-1.0-0-dev
+          - libvdeplug-dev
+          - libvte-2.91-dev
+          # Tests dependencies
+          - genisoimage
+      env:
+        - TEST_CMD="make check-unit"
+        - CONFIG="--disable-containers --disable-tcg --enable-kvm --disable-tools"
+      script:
+        - ( cd ${SRC_DIR} ; git submodule update --init roms/SLOF )
+        - BUILD_RC=0 && make -j${JOBS} || BUILD_RC=$?
+        - |
+          if [ "$BUILD_RC" -eq 0 ] ; then
+              mv pc-bios/s390-ccw/*.img pc-bios/ ;
+              ${TEST_CMD} ;
+          else
+              $(exit $BUILD_RC);
+          fi
+
     # Release builds
     # The make-release script expect a QEMU version, so our tag must start with a 'v'.
     # This is the case when release candidate tags are created.
-- 
2.21.1

Re: [PATCH v2 4/4] .travis.yml: Add a KVM-only s390x job

[Cornelia Huck]

On Wed, 18 Mar 2020 23:27:17 +0100
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:

> Add a job to build QEMU on s390x with TCG disabled, so
> this configuration won't bitrot over time.
> 
> This job is quick, running check-unit: Ran for 4 min 48 sec
> https://travis-ci.org/github/philmd/qemu/jobs/663659486
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  .travis.yml | 42 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 42 insertions(+)

That looks like a good idea.

Acked-by: Cornelia Huck <cohuck@redhat.com>

Re: [PATCH v2 0/4] travis-ci: Add a KVM-only s390x job

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200318222717.24676-1-philmd@redhat.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

  TEST    check-unit: tests/test-rcu-simpleq
  TEST    iotest-qcow2: 024
**
ERROR:/tmp/qemu-test/src/tests/qtest/migration-test.c:1249:test_migrate_auto_converge: assertion failed (remaining < (expected_threshold + expected_threshold / 100)): (103809024 < 101000000)
ERROR - Bail out! ERROR:/tmp/qemu-test/src/tests/qtest/migration-test.c:1249:test_migrate_auto_converge: assertion failed (remaining < (expected_threshold + expected_threshold / 100)): (103809024 < 101000000)
make: *** [check-qtest-aarch64] Error 1
make: *** Waiting for unfinished jobs....
  TEST    check-unit: tests/test-rcu-tailq
  TEST    iotest-qcow2: 025
---
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
**
ERROR:/tmp/qemu-test/src/tests/qtest/migration-test.c:1249:test_migrate_auto_converge: assertion failed (remaining < (expected_threshold + expected_threshold / 100)): (103809024 < 101000000)
ERROR - Bail out! ERROR:/tmp/qemu-test/src/tests/qtest/migration-test.c:1249:test_migrate_auto_converge: assertion failed (remaining < (expected_threshold + expected_threshold / 100)): (103809024 < 101000000)
make: *** [check-qtest-x86_64] Error 1
  TEST    iotest-qcow2: 041
  TEST    iotest-qcow2: 042
  TEST    iotest-qcow2: 043
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=ff24ee2953f1454290305113cefa8f7e', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-qj2x29ta/src/docker-src.2020-03-18-20.12.13.29783:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=ff24ee2953f1454290305113cefa8f7e
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-qj2x29ta/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    11m32.251s
user    0m8.589s


The full log is available at
http://patchew.org/logs/20200318222717.24676-1-philmd@redhat.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com