[PATCH 00/12] testing/next: docker, avocado, unit,

[PATCH 00/12] testing/next: docker, avocado, unit,

[Alex Bennée]

Here is the current status of my testing next. I've had to jump a few
hoops on MacOS as adding socat discovered a unit test failure. This
also includes tuxrun_baselines which is a collection of quick test
images that exercise a wide range of platforms.

The following patches need review:

tests: ensure we export job results for some cross builds
tests: add tuxrun baseline test to avocado
tests: skip the nios2 replay_kernel test
testing: update ubuntu2004 to ubuntu2204
gitlab: extend custom runners with base_job_template
gitlab: reduce default verbosity of cirrus run
tests: be a bit more strict cleaning up fifos
tests: add socat dependency for tests
gitlab: tweak and filter ninja output to reduce build noise

Alex Bennée (10):
  gitlab: tweak and filter ninja output to reduce build noise
  tests/avocado: retire the Aarch64 TCG tests from boot_linux.py
  tests: add socat dependency for tests
  tests: be a bit more strict cleaning up fifos
  gitlab: reduce default verbosity of cirrus run
  gitlab: extend custom runners with base_job_template
  testing: update ubuntu2004 to ubuntu2204
  tests: skip the nios2 replay_kernel test
  tests: add tuxrun baseline test to avocado
  tests: ensure we export job results for some cross builds

Bastian Koppelmann (1):
  tests/docker: Use binaries for debian-tricore-cross

Thomas Huth (1):
  cirrus.yml: Improve the windows_msys2_task

 MAINTAINERS                                   |   1 +
 docs/devel/testing.rst                        |   4 +-
 tests/tcg/tricore/macros.h                    |   2 +-
 tests/unit/test-io-channel-command.c          |  10 +-
 .cirrus.yml                                   |   8 +-
 .gitlab-ci.d/buildtest-template.yml           |   2 +-
 .gitlab-ci.d/buildtest.yml                    |  22 +-
 .gitlab-ci.d/cirrus/build.yml                 |   4 +-
 .gitlab-ci.d/cirrus/freebsd-12.vars           |   2 +-
 .gitlab-ci.d/cirrus/freebsd-13.vars           |   2 +-
 .gitlab-ci.d/cirrus/macos-12.vars             |   2 +-
 .gitlab-ci.d/containers.yml                   |   4 +-
 .gitlab-ci.d/crossbuild-template.yml          |  11 +
 .gitlab-ci.d/crossbuilds.yml                  |  12 +-
 .gitlab-ci.d/custom-runners.yml               |   3 +-
 .../custom-runners/ubuntu-20.04-s390x.yml     |  10 +-
 .../custom-runners/ubuntu-22.04-aarch32.yml   |   2 +-
 .../custom-runners/ubuntu-22.04-aarch64.yml   |  10 +-
 tests/avocado/boot_linux.py                   |  48 +-
 tests/avocado/machine_aarch64_virt.py         |  64 ++-
 tests/avocado/replay_kernel.py                |   1 +
 tests/avocado/tuxrun_baselines.py             | 423 ++++++++++++++++++
 tests/docker/dockerfiles/alpine.docker        |   1 +
 tests/docker/dockerfiles/centos8.docker       |   1 +
 .../dockerfiles/debian-amd64-cross.docker     |   1 +
 tests/docker/dockerfiles/debian-amd64.docker  |   1 +
 .../dockerfiles/debian-arm64-cross.docker     |   1 +
 .../dockerfiles/debian-armel-cross.docker     |   1 +
 .../dockerfiles/debian-armhf-cross.docker     |   1 +
 .../dockerfiles/debian-mips64el-cross.docker  |   1 +
 .../dockerfiles/debian-mipsel-cross.docker    |   1 +
 .../dockerfiles/debian-ppc64el-cross.docker   |   1 +
 .../dockerfiles/debian-s390x-cross.docker     |   1 +
 .../dockerfiles/debian-tricore-cross.docker   |  10 +-
 .../dockerfiles/fedora-win32-cross.docker     |   1 +
 .../dockerfiles/fedora-win64-cross.docker     |   1 +
 tests/docker/dockerfiles/fedora.docker        |   1 +
 tests/docker/dockerfiles/opensuse-leap.docker |   1 +
 .../{ubuntu2004.docker => ubuntu2204.docker}  |  17 +-
 tests/docker/test-tsan                        |   2 +-
 tests/lcitool/projects/qemu.yml               |   1 +
 tests/lcitool/refresh                         |  10 +-
 tests/tcg/tricore/Makefile.softmmu-target     |   6 +-
 43 files changed, 589 insertions(+), 119 deletions(-)
 create mode 100644 tests/avocado/tuxrun_baselines.py
 rename tests/docker/dockerfiles/{ubuntu2004.docker => ubuntu2204.docker} (91%)

-- 
2.39.1

[PATCH 01/12] gitlab: tweak and filter ninja output to reduce build noise

[Alex Bennée]

A significant portion of our CI logs are just enumerating each
successfully built object file. The current widespread versions of
ninja don't have a quiet option so we use NINJA_STATUS to add a fixed
string to the ninja output which we then filter with grep. If there
are any errors in the output we get them from the compiler.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .gitlab-ci.d/buildtest-template.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index 73ecfabb8d..3af51846cd 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -21,7 +21,7 @@
       then
         ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ;
       fi || exit 1;
-    - make -j"$JOBS"
+    - env NINJA_STATUS="[ninja][%f/%t] " make -j"$JOBS" | grep -v "\[ninja\]\[.*[123456789]/"
     - if test -n "$MAKE_CHECK_ARGS";
       then
         make -j"$JOBS" $MAKE_CHECK_ARGS ;
-- 
2.39.1

Re: [PATCH 01/12] gitlab: tweak and filter ninja output to reduce build noise

[Thomas Huth]

On 15/02/2023 20.25, Alex Bennée wrote:
> A significant portion of our CI logs are just enumerating each
> successfully built object file. The current widespread versions of
> ninja don't have a quiet option so we use NINJA_STATUS to add a fixed
> string to the ninja output which we then filter with grep. If there
> are any errors in the output we get them from the compiler.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   .gitlab-ci.d/buildtest-template.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
> index 73ecfabb8d..3af51846cd 100644
> --- a/.gitlab-ci.d/buildtest-template.yml
> +++ b/.gitlab-ci.d/buildtest-template.yml
> @@ -21,7 +21,7 @@
>         then
>           ../meson/meson.py configure . -Dbackend_max_links="$LD_JOBS" ;
>         fi || exit 1;
> -    - make -j"$JOBS"
> +    - env NINJA_STATUS="[ninja][%f/%t] " make -j"$JOBS" | grep -v "\[ninja\]\[.*[123456789]/"
>       - if test -n "$MAKE_CHECK_ARGS";
>         then
>           make -j"$JOBS" $MAKE_CHECK_ARGS ;

Not meant as a veto, but just for the records: I still don't like the idea. 
Having a log of the files that got compiled is still sometimes useful for 
me, e.g. when I want to check whether a certain file has been compiled at 
all or not (when e.g. debugging meson.build problems). So I'm still in 
favour of dropping this patch.

IMHO if you want to shorten the build log in the CI, please get those chatty 
softfloat tests fixed instead.

  Thomas

[PATCH 02/12] tests/avocado: retire the Aarch64 TCG tests from boot_linux.py

[Alex Bennée]

The two TCG tests for GICv2 and GICv3 are very heavy weight distros
that take a long time to boot up, especially for an --enable-debug
build. The total code coverage they give is:

  Overall coverage rate:
    lines......: 11.2% (59584 of 530123 lines)
    functions..: 15.0% (7436 of 49443 functions)
    branches...: 6.3% (19273 of 303933 branches)

We already get pretty close to that with the machine_aarch64_virt
tests which only does one full boot (~120s vs ~600s) of alpine. We
expand the kernel+initrd boot (~8s) to test both GICs and also add an
RNG device and a block device to generate a few IRQs and exercise the
storage layer. With that we get to a coverage of:

  Overall coverage rate:
    lines......: 11.0% (58121 of 530123 lines)
    functions..: 14.9% (7343 of 49443 functions)
    branches...: 6.0% (18269 of 303933 branches)

which I feel is close enough given the massive time saving. If we want
to target any more sub-systems we can use lighter weight more directed
tests.

Reviewed-by: Fabiano Rosas <farosas@suse.de>
Acked-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Peter Maydell <peter.maydell@linaro.org>

---
v2
  - fix checkpatch warning
---
 tests/avocado/boot_linux.py           | 48 ++++----------------
 tests/avocado/machine_aarch64_virt.py | 64 ++++++++++++++++++++++++---
 2 files changed, 66 insertions(+), 46 deletions(-)

diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py
index b3e58fa309..fe0bb180d9 100644
--- a/tests/avocado/boot_linux.py
+++ b/tests/avocado/boot_linux.py
@@ -58,52 +58,16 @@ def test_pc_q35_kvm(self):
         self.launch_and_wait(set_up_ssh_connection=False)
 
 
-# For Aarch64 we only boot KVM tests in CI as the TCG tests are very
-# heavyweight. There are lighter weight distros which we use in the
-# machine_aarch64_virt.py tests.
+# For Aarch64 we only boot KVM tests in CI as booting the current
+# Fedora OS in TCG tests is very heavyweight. There are lighter weight
+# distros which we use in the machine_aarch64_virt.py tests.
 class BootLinuxAarch64(LinuxTest):
     """
     :avocado: tags=arch:aarch64
     :avocado: tags=machine:virt
-    :avocado: tags=machine:gic-version=2
     """
     timeout = 720
 
-    def add_common_args(self):
-        self.vm.add_args('-bios',
-                         os.path.join(BUILD_DIR, 'pc-bios',
-                                      'edk2-aarch64-code.fd'))
-        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
-        self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
-
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
-    def test_fedora_cloud_tcg_gicv2(self):
-        """
-        :avocado: tags=accel:tcg
-        :avocado: tags=cpu:max
-        :avocado: tags=device:gicv2
-        """
-        self.require_accelerator("tcg")
-        self.vm.add_args("-accel", "tcg")
-        self.vm.add_args("-cpu", "max,lpa2=off")
-        self.vm.add_args("-machine", "virt,gic-version=2")
-        self.add_common_args()
-        self.launch_and_wait(set_up_ssh_connection=False)
-
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
-    def test_fedora_cloud_tcg_gicv3(self):
-        """
-        :avocado: tags=accel:tcg
-        :avocado: tags=cpu:max
-        :avocado: tags=device:gicv3
-        """
-        self.require_accelerator("tcg")
-        self.vm.add_args("-accel", "tcg")
-        self.vm.add_args("-cpu", "max,lpa2=off")
-        self.vm.add_args("-machine", "virt,gic-version=3")
-        self.add_common_args()
-        self.launch_and_wait(set_up_ssh_connection=False)
-
     def test_virt_kvm(self):
         """
         :avocado: tags=accel:kvm
@@ -112,7 +76,11 @@ def test_virt_kvm(self):
         self.require_accelerator("kvm")
         self.vm.add_args("-accel", "kvm")
         self.vm.add_args("-machine", "virt,gic-version=host")
-        self.add_common_args()
+        self.vm.add_args('-bios',
+                         os.path.join(BUILD_DIR, 'pc-bios',
+                                      'edk2-aarch64-code.fd'))
+        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
+        self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
         self.launch_and_wait(set_up_ssh_connection=False)
 
 
diff --git a/tests/avocado/machine_aarch64_virt.py b/tests/avocado/machine_aarch64_virt.py
index c2b2ba2cf8..60921fed8b 100644
--- a/tests/avocado/machine_aarch64_virt.py
+++ b/tests/avocado/machine_aarch64_virt.py
@@ -10,11 +10,14 @@
 
 import time
 import os
+import logging
 
 from avocado_qemu import QemuSystemTest
 from avocado_qemu import wait_for_console_pattern
 from avocado_qemu import exec_command
 from avocado_qemu import BUILD_DIR
+from avocado.utils import process
+from avocado.utils.path import find_command
 
 class Aarch64VirtMachine(QemuSystemTest):
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
@@ -65,16 +68,15 @@ def test_alpine_virt_tcg_gic_max(self):
         self.wait_for_console_pattern('Welcome to Alpine Linux 3.16')
 
 
-    def test_aarch64_virt(self):
+    def common_aarch64_virt(self, machine):
         """
-        :avocado: tags=arch:aarch64
-        :avocado: tags=machine:virt
-        :avocado: tags=accel:tcg
-        :avocado: tags=cpu:max
+        Common code to launch basic virt machine with kernel+initrd
+        and a scratch disk.
         """
+        logger = logging.getLogger('aarch64_virt')
+
         kernel_url = ('https://fileserver.linaro.org/s/'
                       'z6B2ARM7DQT3HWN/download')
-
         kernel_hash = 'ed11daab50c151dde0e1e9c9cb8b2d9bd3215347'
         kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
 
@@ -83,13 +85,63 @@ def test_aarch64_virt(self):
                                'console=ttyAMA0')
         self.require_accelerator("tcg")
         self.vm.add_args('-cpu', 'max,pauth-impdef=on',
+                         '-machine', machine,
                          '-accel', 'tcg',
                          '-kernel', kernel_path,
                          '-append', kernel_command_line)
+
+        # A RNG offers an easy way to generate a few IRQs
+        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
+        self.vm.add_args('-object',
+                         'rng-random,id=rng0,filename=/dev/urandom')
+
+        # Also add a scratch block device
+        logger.info('creating scratch qcow2 image')
+        image_path = os.path.join(self.workdir, 'scratch.qcow2')
+        qemu_img = os.path.join(BUILD_DIR, 'qemu-img')
+        if not os.path.exists(qemu_img):
+            qemu_img = find_command('qemu-img', False)
+        if qemu_img is False:
+            self.cancel('Could not find "qemu-img", which is required to '
+                        'create the temporary qcow2 image')
+        cmd = '%s create -f qcow2 %s 8M' % (qemu_img, image_path)
+        process.run(cmd)
+
+        # Add the device
+        self.vm.add_args('-blockdev',
+                         f"driver=qcow2,file.driver=file,"
+                         f"file.filename={image_path},node-name=scratch")
+        self.vm.add_args('-device',
+                         'virtio-blk-device,drive=scratch')
+
         self.vm.launch()
         self.wait_for_console_pattern('Welcome to Buildroot')
         time.sleep(0.1)
         exec_command(self, 'root')
         time.sleep(0.1)
+        exec_command(self, 'dd if=/dev/hwrng of=/dev/vda bs=512 count=4')
+        time.sleep(0.1)
+        exec_command(self, 'md5sum /dev/vda')
+        time.sleep(0.1)
+        exec_command(self, 'cat /proc/interrupts')
+        time.sleep(0.1)
         exec_command(self, 'cat /proc/self/maps')
         time.sleep(0.1)
+
+    def test_aarch64_virt_gicv3(self):
+        """
+        :avocado: tags=arch:aarch64
+        :avocado: tags=machine:virt
+        :avocado: tags=accel:tcg
+        :avocado: tags=cpu:max
+        """
+        self.common_aarch64_virt("virt,gic_version=3")
+
+    def test_aarch64_virt_gicv2(self):
+        """
+        :avocado: tags=arch:aarch64
+        :avocado: tags=machine:virt
+        :avocado: tags=accel:tcg
+        :avocado: tags=cpu:max
+        """
+        self.common_aarch64_virt("virt,gic-version=2")
-- 
2.39.1

Re: [PATCH 02/12] tests/avocado: retire the Aarch64 TCG tests from boot_linux.py

[Philippe Mathieu-Daudé]

On 15/2/23 20:25, Alex Bennée wrote:
> The two TCG tests for GICv2 and GICv3 are very heavy weight distros
> that take a long time to boot up, especially for an --enable-debug
> build. The total code coverage they give is:
> 
>    Overall coverage rate:
>      lines......: 11.2% (59584 of 530123 lines)
>      functions..: 15.0% (7436 of 49443 functions)
>      branches...: 6.3% (19273 of 303933 branches)
> 
> We already get pretty close to that with the machine_aarch64_virt
> tests which only does one full boot (~120s vs ~600s) of alpine. We
> expand the kernel+initrd boot (~8s) to test both GICs and also add an
> RNG device and a block device to generate a few IRQs and exercise the
> storage layer. With that we get to a coverage of:
> 
>    Overall coverage rate:
>      lines......: 11.0% (58121 of 530123 lines)
>      functions..: 14.9% (7343 of 49443 functions)
>      branches...: 6.0% (18269 of 303933 branches)
> 
> which I feel is close enough given the massive time saving. If we want
> to target any more sub-systems we can use lighter weight more directed
> tests.
> 
> Reviewed-by: Fabiano Rosas <farosas@suse.de>
> Acked-by: Richard Henderson <richard.henderson@linaro.org>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> 
> ---
> v2
>    - fix checkpatch warning
> ---
>   tests/avocado/boot_linux.py           | 48 ++++----------------
>   tests/avocado/machine_aarch64_virt.py | 64 ++++++++++++++++++++++++---
>   2 files changed, 66 insertions(+), 46 deletions(-)


> -    def test_aarch64_virt(self):
> +    def common_aarch64_virt(self, machine):
>           """
> -        :avocado: tags=arch:aarch64
> -        :avocado: tags=machine:virt
> -        :avocado: tags=accel:tcg
> -        :avocado: tags=cpu:max
> +        Common code to launch basic virt machine with kernel+initrd
> +        and a scratch disk.
>           """
> +        logger = logging.getLogger('aarch64_virt')
> +
>           kernel_url = ('https://fileserver.linaro.org/s/'
>                         'z6B2ARM7DQT3HWN/download')
> -
>           kernel_hash = 'ed11daab50c151dde0e1e9c9cb8b2d9bd3215347'
>           kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>   
> @@ -83,13 +85,63 @@ def test_aarch64_virt(self):
>                                  'console=ttyAMA0')
>           self.require_accelerator("tcg")
>           self.vm.add_args('-cpu', 'max,pauth-impdef=on',
> +                         '-machine', machine,

I'm surprised you had to add that, this should be handled by
avocado_qemu::get_vm which calls QEMUMachine::set_machine:

     def get_vm(self, *args, name=None):
             if self.machine is not None:
                 self._vms[name].set_machine(self.machine)

>                            '-accel', 'tcg',
>                            '-kernel', kernel_path,
>                            '-append', kernel_command_line)
> +
> +        # A RNG offers an easy way to generate a few IRQs
> +        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
> +        self.vm.add_args('-object',
> +                         'rng-random,id=rng0,filename=/dev/urandom')
> +
> +        # Also add a scratch block device
> +        logger.info('creating scratch qcow2 image')
> +        image_path = os.path.join(self.workdir, 'scratch.qcow2')
> +        qemu_img = os.path.join(BUILD_DIR, 'qemu-img')
> +        if not os.path.exists(qemu_img):
> +            qemu_img = find_command('qemu-img', False)
> +        if qemu_img is False:
> +            self.cancel('Could not find "qemu-img", which is required to '
> +                        'create the temporary qcow2 image')
> +        cmd = '%s create -f qcow2 %s 8M' % (qemu_img, image_path)
> +        process.run(cmd)
> +
> +        # Add the device
> +        self.vm.add_args('-blockdev',
> +                         f"driver=qcow2,file.driver=file,"
> +                         f"file.filename={image_path},node-name=scratch")
> +        self.vm.add_args('-device',
> +                         'virtio-blk-device,drive=scratch')
> +
>           self.vm.launch()
>           self.wait_for_console_pattern('Welcome to Buildroot')
>           time.sleep(0.1)
>           exec_command(self, 'root')
>           time.sleep(0.1)
> +        exec_command(self, 'dd if=/dev/hwrng of=/dev/vda bs=512 count=4')
> +        time.sleep(0.1)
> +        exec_command(self, 'md5sum /dev/vda')
> +        time.sleep(0.1)
> +        exec_command(self, 'cat /proc/interrupts')
> +        time.sleep(0.1)
>           exec_command(self, 'cat /proc/self/maps')
>           time.sleep(0.1)
> +
> +    def test_aarch64_virt_gicv3(self):
> +        """
> +        :avocado: tags=arch:aarch64
> +        :avocado: tags=machine:virt

(machine set here)

> +        :avocado: tags=accel:tcg
> +        :avocado: tags=cpu:max
> +        """
> +        self.common_aarch64_virt("virt,gic_version=3")

Ah, you duplicate as "-machine virt -machine virt,gic_version=3".

It would be clearer if you rename common_aarch64_virt() argument
as 'machine_options' and pass the option as:

          """
          :avocado: tags=machine:virt
          """
          self.common_aarch64_virt("gic_version=3")

to produce "-machine virt -machine gic_version=3".

Anyhow,
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

[PATCH 03/12] tests: add socat dependency for tests

[Alex Bennée]

We only use it for test-io-channel-command at the moment.
Unfortunately bringing socat into CI exposes an existing bug in the
test-io-channel-command unit test. For now disable the test with the
preprocessor until someone can diagnose it on Mac hardware.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
---
 tests/unit/test-io-channel-command.c                  | 4 ++--
 .gitlab-ci.d/cirrus/freebsd-12.vars                   | 2 +-
 .gitlab-ci.d/cirrus/freebsd-13.vars                   | 2 +-
 .gitlab-ci.d/cirrus/macos-12.vars                     | 2 +-
 tests/docker/dockerfiles/alpine.docker                | 1 +
 tests/docker/dockerfiles/centos8.docker               | 1 +
 tests/docker/dockerfiles/debian-amd64-cross.docker    | 1 +
 tests/docker/dockerfiles/debian-amd64.docker          | 1 +
 tests/docker/dockerfiles/debian-arm64-cross.docker    | 1 +
 tests/docker/dockerfiles/debian-armel-cross.docker    | 1 +
 tests/docker/dockerfiles/debian-armhf-cross.docker    | 1 +
 tests/docker/dockerfiles/debian-mips64el-cross.docker | 1 +
 tests/docker/dockerfiles/debian-mipsel-cross.docker   | 1 +
 tests/docker/dockerfiles/debian-ppc64el-cross.docker  | 1 +
 tests/docker/dockerfiles/debian-s390x-cross.docker    | 1 +
 tests/docker/dockerfiles/fedora-win32-cross.docker    | 1 +
 tests/docker/dockerfiles/fedora-win64-cross.docker    | 1 +
 tests/docker/dockerfiles/fedora.docker                | 1 +
 tests/docker/dockerfiles/opensuse-leap.docker         | 1 +
 tests/docker/dockerfiles/ubuntu2004.docker            | 1 +
 tests/lcitool/projects/qemu.yml                       | 1 +
 21 files changed, 22 insertions(+), 5 deletions(-)

diff --git a/tests/unit/test-io-channel-command.c b/tests/unit/test-io-channel-command.c
index 425e2f5594..f3c34152ac 100644
--- a/tests/unit/test-io-channel-command.c
+++ b/tests/unit/test-io-channel-command.c
@@ -31,7 +31,7 @@
 
 static char *socat = NULL;
 
-#ifndef _WIN32
+#if !defined(_WIN32) && !defined(DARWIN)
 static void test_io_channel_command_fifo(bool async)
 {
     g_autofree gchar *tmpdir = g_dir_make_tmp("qemu-test-io-channel.XXXXXX", NULL);
@@ -128,7 +128,7 @@ int main(int argc, char **argv)
 
     socat = g_find_program_in_path("socat");
 
-#ifndef _WIN32
+#if !defined(_WIN32) && !defined(DARWIN)
     g_test_add_func("/io/channel/command/fifo/sync",
                     test_io_channel_command_fifo_sync);
     g_test_add_func("/io/channel/command/fifo/async",
diff --git a/.gitlab-ci.d/cirrus/freebsd-12.vars b/.gitlab-ci.d/cirrus/freebsd-12.vars
index 8934e5d57f..44d8a2a511 100644
--- a/.gitlab-ci.d/cirrus/freebsd-12.vars
+++ b/.gitlab-ci.d/cirrus/freebsd-12.vars
@@ -11,6 +11,6 @@ MAKE='/usr/local/bin/gmake'
 NINJA='/usr/local/bin/ninja'
 PACKAGING_COMMAND='pkg'
 PIP3='/usr/local/bin/pip-3.8'
-PKGS='alsa-lib bash bison bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage cmocka ctags curl cyrus-sasl dbus diffutils dtc flex fusefs-libs3 gettext git glib gmake gnutls gsed gtk3 json-c libepoxy libffi libgcrypt libjpeg-turbo libnfs libslirp libspice-server libssh libtasn1 llvm lzo2 meson ncurses nettle ninja opencv pixman pkgconf png py39-numpy py39-pillow py39-pip py39-sphinx py39-sphinx_rtd_theme py39-yaml python3 rpm2cpio sdl2 sdl2_image snappy sndio spice-protocol tesseract usbredir virglrenderer vte3 zstd'
+PKGS='alsa-lib bash bison bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage cmocka ctags curl cyrus-sasl dbus diffutils dtc flex fusefs-libs3 gettext git glib gmake gnutls gsed gtk3 json-c libepoxy libffi libgcrypt libjpeg-turbo libnfs libslirp libspice-server libssh libtasn1 llvm lzo2 meson ncurses nettle ninja opencv pixman pkgconf png py39-numpy py39-pillow py39-pip py39-sphinx py39-sphinx_rtd_theme py39-yaml python3 rpm2cpio sdl2 sdl2_image snappy sndio socat spice-protocol tesseract usbredir virglrenderer vte3 zstd'
 PYPI_PKGS=''
 PYTHON='/usr/local/bin/python3'
diff --git a/.gitlab-ci.d/cirrus/freebsd-13.vars b/.gitlab-ci.d/cirrus/freebsd-13.vars
index 65ce456c48..7622c849b2 100644
--- a/.gitlab-ci.d/cirrus/freebsd-13.vars
+++ b/.gitlab-ci.d/cirrus/freebsd-13.vars
@@ -11,6 +11,6 @@ MAKE='/usr/local/bin/gmake'
 NINJA='/usr/local/bin/ninja'
 PACKAGING_COMMAND='pkg'
 PIP3='/usr/local/bin/pip-3.8'
-PKGS='alsa-lib bash bison bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage cmocka ctags curl cyrus-sasl dbus diffutils dtc flex fusefs-libs3 gettext git glib gmake gnutls gsed gtk3 json-c libepoxy libffi libgcrypt libjpeg-turbo libnfs libslirp libspice-server libssh libtasn1 llvm lzo2 meson ncurses nettle ninja opencv pixman pkgconf png py39-numpy py39-pillow py39-pip py39-sphinx py39-sphinx_rtd_theme py39-yaml python3 rpm2cpio sdl2 sdl2_image snappy sndio spice-protocol tesseract usbredir virglrenderer vte3 zstd'
+PKGS='alsa-lib bash bison bzip2 ca_root_nss capstone4 ccache cdrkit-genisoimage cmocka ctags curl cyrus-sasl dbus diffutils dtc flex fusefs-libs3 gettext git glib gmake gnutls gsed gtk3 json-c libepoxy libffi libgcrypt libjpeg-turbo libnfs libslirp libspice-server libssh libtasn1 llvm lzo2 meson ncurses nettle ninja opencv pixman pkgconf png py39-numpy py39-pillow py39-pip py39-sphinx py39-sphinx_rtd_theme py39-yaml python3 rpm2cpio sdl2 sdl2_image snappy sndio socat spice-protocol tesseract usbredir virglrenderer vte3 zstd'
 PYPI_PKGS=''
 PYTHON='/usr/local/bin/python3'
diff --git a/.gitlab-ci.d/cirrus/macos-12.vars b/.gitlab-ci.d/cirrus/macos-12.vars
index 65b78fa08f..da6aa6469b 100644
--- a/.gitlab-ci.d/cirrus/macos-12.vars
+++ b/.gitlab-ci.d/cirrus/macos-12.vars
@@ -11,6 +11,6 @@ MAKE='/opt/homebrew/bin/gmake'
 NINJA='/opt/homebrew/bin/ninja'
 PACKAGING_COMMAND='brew'
 PIP3='/opt/homebrew/bin/pip3'
-PKGS='bash bc bison bzip2 capstone ccache cmocka ctags curl dbus diffutils dtc flex gcovr gettext git glib gnu-sed gnutls gtk+3 jemalloc jpeg-turbo json-c libepoxy libffi libgcrypt libiscsi libnfs libpng libslirp libssh libtasn1 libusb llvm lzo make meson ncurses nettle ninja pixman pkg-config python3 rpm2cpio sdl2 sdl2_image snappy sparse spice-protocol tesseract usbredir vde vte3 zlib zstd'
+PKGS='bash bc bison bzip2 capstone ccache cmocka ctags curl dbus diffutils dtc flex gcovr gettext git glib gnu-sed gnutls gtk+3 jemalloc jpeg-turbo json-c libepoxy libffi libgcrypt libiscsi libnfs libpng libslirp libssh libtasn1 libusb llvm lzo make meson ncurses nettle ninja pixman pkg-config python3 rpm2cpio sdl2 sdl2_image snappy socat sparse spice-protocol tesseract usbredir vde vte3 zlib zstd'
 PYPI_PKGS='PyYAML numpy pillow sphinx sphinx-rtd-theme'
 PYTHON='/opt/homebrew/bin/python3'
diff --git a/tests/docker/dockerfiles/alpine.docker b/tests/docker/dockerfiles/alpine.docker
index 4a569d82f6..56cf14e553 100644
--- a/tests/docker/dockerfiles/alpine.docker
+++ b/tests/docker/dockerfiles/alpine.docker
@@ -94,6 +94,7 @@ RUN apk update && \
         sed \
         snappy-dev \
         sndio-dev \
+        socat \
         sparse \
         spice-dev \
         spice-protocol \
diff --git a/tests/docker/dockerfiles/centos8.docker b/tests/docker/dockerfiles/centos8.docker
index fbc953c6dc..1291ae1b04 100644
--- a/tests/docker/dockerfiles/centos8.docker
+++ b/tests/docker/dockerfiles/centos8.docker
@@ -105,6 +105,7 @@ RUN dnf distro-sync -y && \
         rpm \
         sed \
         snappy-devel \
+        socat \
         spice-protocol \
         spice-server-devel \
         systemd-devel \
diff --git a/tests/docker/dockerfiles/debian-amd64-cross.docker b/tests/docker/dockerfiles/debian-amd64-cross.docker
index 5175095a85..856db95100 100644
--- a/tests/docker/dockerfiles/debian-amd64-cross.docker
+++ b/tests/docker/dockerfiles/debian-amd64-cross.docker
@@ -52,6 +52,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       python3-yaml \
                       rpm2cpio \
                       sed \
+                      socat \
                       sparse \
                       tar \
                       tesseract-ocr \
diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
index b61f664ea2..e3dba71ad5 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -120,6 +120,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       python3-yaml \
                       rpm2cpio \
                       sed \
+                      socat \
                       sparse \
                       systemtap-sdt-dev \
                       tar \
diff --git a/tests/docker/dockerfiles/debian-arm64-cross.docker b/tests/docker/dockerfiles/debian-arm64-cross.docker
index b69958c69f..b00e9e9bcf 100644
--- a/tests/docker/dockerfiles/debian-arm64-cross.docker
+++ b/tests/docker/dockerfiles/debian-arm64-cross.docker
@@ -52,6 +52,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       python3-yaml \
                       rpm2cpio \
                       sed \
+                      socat \
                       sparse \
                       tar \
                       tesseract-ocr \
diff --git a/tests/docker/dockerfiles/debian-armel-cross.docker b/tests/docker/dockerfiles/debian-armel-cross.docker
index 96b524fab6..fb1129f256 100644
--- a/tests/docker/dockerfiles/debian-armel-cross.docker
+++ b/tests/docker/dockerfiles/debian-armel-cross.docker
@@ -52,6 +52,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       python3-yaml \
                       rpm2cpio \
                       sed \
+                      socat \
                       sparse \
                       tar \
                       tesseract-ocr \
diff --git a/tests/docker/dockerfiles/debian-armhf-cross.docker b/tests/docker/dockerfiles/debian-armhf-cross.docker
index 08a75cebdb..7a2b864a38 100644
--- a/tests/docker/dockerfiles/debian-armhf-cross.docker
+++ b/tests/docker/dockerfiles/debian-armhf-cross.docker
@@ -52,6 +52,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       python3-yaml \
                       rpm2cpio \
                       sed \
+                      socat \
                       sparse \
                       tar \
                       tesseract-ocr \
diff --git a/tests/docker/dockerfiles/debian-mips64el-cross.docker b/tests/docker/dockerfiles/debian-mips64el-cross.docker
index 5930e6fa5d..5a3340e964 100644
--- a/tests/docker/dockerfiles/debian-mips64el-cross.docker
+++ b/tests/docker/dockerfiles/debian-mips64el-cross.docker
@@ -52,6 +52,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       python3-yaml \
                       rpm2cpio \
                       sed \
+                      socat \
                       sparse \
                       tar \
                       tesseract-ocr \
diff --git a/tests/docker/dockerfiles/debian-mipsel-cross.docker b/tests/docker/dockerfiles/debian-mipsel-cross.docker
index c65d9830e7..422fdebe8f 100644
--- a/tests/docker/dockerfiles/debian-mipsel-cross.docker
+++ b/tests/docker/dockerfiles/debian-mipsel-cross.docker
@@ -52,6 +52,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       python3-yaml \
                       rpm2cpio \
                       sed \
+                      socat \
                       sparse \
                       tar \
                       tesseract-ocr \
diff --git a/tests/docker/dockerfiles/debian-ppc64el-cross.docker b/tests/docker/dockerfiles/debian-ppc64el-cross.docker
index 2ae56c978e..78d7ae6211 100644
--- a/tests/docker/dockerfiles/debian-ppc64el-cross.docker
+++ b/tests/docker/dockerfiles/debian-ppc64el-cross.docker
@@ -52,6 +52,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       python3-yaml \
                       rpm2cpio \
                       sed \
+                      socat \
                       sparse \
                       tar \
                       tesseract-ocr \
diff --git a/tests/docker/dockerfiles/debian-s390x-cross.docker b/tests/docker/dockerfiles/debian-s390x-cross.docker
index 0db86a0fcd..d06ea3605a 100644
--- a/tests/docker/dockerfiles/debian-s390x-cross.docker
+++ b/tests/docker/dockerfiles/debian-s390x-cross.docker
@@ -52,6 +52,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       python3-yaml \
                       rpm2cpio \
                       sed \
+                      socat \
                       sparse \
                       tar \
                       tesseract-ocr \
diff --git a/tests/docker/dockerfiles/fedora-win32-cross.docker b/tests/docker/dockerfiles/fedora-win32-cross.docker
index b659c0b8a8..21ed1c6081 100644
--- a/tests/docker/dockerfiles/fedora-win32-cross.docker
+++ b/tests/docker/dockerfiles/fedora-win32-cross.docker
@@ -52,6 +52,7 @@ exec "$@"' > /usr/bin/nosync && \
                python3-sphinx_rtd_theme \
                rpm \
                sed \
+               socat \
                sparse \
                spice-protocol \
                tar \
diff --git a/tests/docker/dockerfiles/fedora-win64-cross.docker b/tests/docker/dockerfiles/fedora-win64-cross.docker
index 0a404c15bf..95d30e7936 100644
--- a/tests/docker/dockerfiles/fedora-win64-cross.docker
+++ b/tests/docker/dockerfiles/fedora-win64-cross.docker
@@ -52,6 +52,7 @@ exec "$@"' > /usr/bin/nosync && \
                python3-sphinx_rtd_theme \
                rpm \
                sed \
+               socat \
                sparse \
                spice-protocol \
                tar \
diff --git a/tests/docker/dockerfiles/fedora.docker b/tests/docker/dockerfiles/fedora.docker
index 5d60a96141..8e06d080b8 100644
--- a/tests/docker/dockerfiles/fedora.docker
+++ b/tests/docker/dockerfiles/fedora.docker
@@ -113,6 +113,7 @@ exec "$@"' > /usr/bin/nosync && \
                rpm \
                sed \
                snappy-devel \
+               socat \
                sparse \
                spice-protocol \
                spice-server-devel \
diff --git a/tests/docker/dockerfiles/opensuse-leap.docker b/tests/docker/dockerfiles/opensuse-leap.docker
index 4b2c02d6ab..568c1c979f 100644
--- a/tests/docker/dockerfiles/opensuse-leap.docker
+++ b/tests/docker/dockerfiles/opensuse-leap.docker
@@ -104,6 +104,7 @@ RUN zypper update -y && \
            sed \
            snappy-devel \
            sndio-devel \
+           socat \
            sparse \
            spice-protocol-devel \
            systemd-devel \
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
index 13ab0b6887..f34d88d33d 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -119,6 +119,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       python3-yaml \
                       rpm2cpio \
                       sed \
+                      socat \
                       sparse \
                       systemtap-sdt-dev \
                       tar \
diff --git a/tests/lcitool/projects/qemu.yml b/tests/lcitool/projects/qemu.yml
index 6467bcf08a..2854748f49 100644
--- a/tests/lcitool/projects/qemu.yml
+++ b/tests/lcitool/projects/qemu.yml
@@ -101,6 +101,7 @@ packages:
  - sed
  - snappy
  - sndio
+ - socat
  - sparse
  - spice-protocol
  - spice-server
-- 
2.39.1

Re: [PATCH 03/12] tests: add socat dependency for tests

[Philippe Mathieu-Daudé]

On 15/2/23 20:25, Alex Bennée wrote:
> We only use it for test-io-channel-command at the moment.
> Unfortunately bringing socat into CI exposes an existing bug in the
> test-io-channel-command unit test. For now disable the test with the
> preprocessor until someone can diagnose it on Mac hardware.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
> Cc: Marc-André Lureau <marcandre.lureau@redhat.com>
> ---
>   tests/unit/test-io-channel-command.c                  | 4 ++--
>   .gitlab-ci.d/cirrus/freebsd-12.vars                   | 2 +-
>   .gitlab-ci.d/cirrus/freebsd-13.vars                   | 2 +-
>   .gitlab-ci.d/cirrus/macos-12.vars                     | 2 +-
>   tests/docker/dockerfiles/alpine.docker                | 1 +
>   tests/docker/dockerfiles/centos8.docker               | 1 +
>   tests/docker/dockerfiles/debian-amd64-cross.docker    | 1 +
>   tests/docker/dockerfiles/debian-amd64.docker          | 1 +
>   tests/docker/dockerfiles/debian-arm64-cross.docker    | 1 +
>   tests/docker/dockerfiles/debian-armel-cross.docker    | 1 +
>   tests/docker/dockerfiles/debian-armhf-cross.docker    | 1 +
>   tests/docker/dockerfiles/debian-mips64el-cross.docker | 1 +
>   tests/docker/dockerfiles/debian-mipsel-cross.docker   | 1 +
>   tests/docker/dockerfiles/debian-ppc64el-cross.docker  | 1 +
>   tests/docker/dockerfiles/debian-s390x-cross.docker    | 1 +
>   tests/docker/dockerfiles/fedora-win32-cross.docker    | 1 +
>   tests/docker/dockerfiles/fedora-win64-cross.docker    | 1 +
>   tests/docker/dockerfiles/fedora.docker                | 1 +
>   tests/docker/dockerfiles/opensuse-leap.docker         | 1 +
>   tests/docker/dockerfiles/ubuntu2004.docker            | 1 +
>   tests/lcitool/projects/qemu.yml                       | 1 +
>   21 files changed, 22 insertions(+), 5 deletions(-)
> 
> diff --git a/tests/unit/test-io-channel-command.c b/tests/unit/test-io-channel-command.c
> index 425e2f5594..f3c34152ac 100644
> --- a/tests/unit/test-io-channel-command.c
> +++ b/tests/unit/test-io-channel-command.c
> @@ -31,7 +31,7 @@
>   
>   static char *socat = NULL;
>   
> -#ifndef _WIN32
> +#if !defined(_WIN32) && !defined(DARWIN)

s/DARWIN/CONFIG_DARWIN/

>   static void test_io_channel_command_fifo(bool async)
>   {
>       g_autofree gchar *tmpdir = g_dir_make_tmp("qemu-test-io-channel.XXXXXX", NULL);
> @@ -128,7 +128,7 @@ int main(int argc, char **argv)
>   
>       socat = g_find_program_in_path("socat");
>   
> -#ifndef _WIN32
> +#if !defined(_WIN32) && !defined(DARWIN)

CONFIG_DARWIN

>       g_test_add_func("/io/channel/command/fifo/sync",
>                       test_io_channel_command_fifo_sync);
>       g_test_add_func("/io/channel/command/fifo/async",

^ Clearly another preliminary patch.

With the split, for both patches, using CONFIG_DARWIN:
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>

[PATCH 04/12] tests: be a bit more strict cleaning up fifos

[Alex Bennée]

When we re-factored we dropped the unlink() step which turns out to be
required for rmdir to do its thing. If we had been checking the return
value we would have noticed so lets do that with this fix.

Fixes: 68406d1085 (tests/unit: cleanups for test-io-channel-command)
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Suggested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 tests/unit/test-io-channel-command.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tests/unit/test-io-channel-command.c b/tests/unit/test-io-channel-command.c
index f3c34152ac..70e82e0e12 100644
--- a/tests/unit/test-io-channel-command.c
+++ b/tests/unit/test-io-channel-command.c
@@ -42,6 +42,7 @@ static void test_io_channel_command_fifo(bool async)
     g_auto(GStrv) dstargv = g_strsplit(dstargs, " ", -1);
     QIOChannel *src, *dst;
     QIOChannelTest *test;
+    int err;
 
     if (mkfifo(fifo, 0600)) {
         g_error("mkfifo: %s", strerror(errno));
@@ -61,7 +62,10 @@ static void test_io_channel_command_fifo(bool async)
     object_unref(OBJECT(src));
     object_unref(OBJECT(dst));
 
-    g_rmdir(tmpdir);
+    err = g_unlink(fifo);
+    g_assert(err == 0);
+    err = g_rmdir(tmpdir);
+    g_assert(err == 0);
 }
 
 static void test_io_channel_command_fifo_async(void)
-- 
2.39.1

Re: [PATCH 04/12] tests: be a bit more strict cleaning up fifos

[Philippe Mathieu-Daudé]

On 15/2/23 20:25, Alex Bennée wrote:
> When we re-factored we dropped the unlink() step which turns out to be
> required for rmdir to do its thing. If we had been checking the return
> value we would have noticed so lets do that with this fix.
> 
> Fixes: 68406d1085 (tests/unit: cleanups for test-io-channel-command)
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Suggested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   tests/unit/test-io-channel-command.c | 6 +++++-
>   1 file changed, 5 insertions(+), 1 deletion(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH 04/12] tests: be a bit more strict cleaning up fifos

[Richard Henderson]

On 2/15/23 09:25, Alex Bennée wrote:
> When we re-factored we dropped the unlink() step which turns out to be
> required for rmdir to do its thing. If we had been checking the return
> value we would have noticed so lets do that with this fix.
> 
> Fixes: 68406d1085 (tests/unit: cleanups for test-io-channel-command)
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
> Suggested-by: Philippe Mathieu-Daudé<philmd@linaro.org>
> ---
>   tests/unit/test-io-channel-command.c | 6 +++++-
>   1 file changed, 5 insertions(+), 1 deletion(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [PATCH 04/12] tests: be a bit more strict cleaning up fifos

[Thomas Huth]

On 15/02/2023 20.25, Alex Bennée wrote:
> When we re-factored we dropped the unlink() step which turns out to be
> required for rmdir to do its thing. If we had been checking the return
> value we would have noticed so lets do that with this fix.
> 
> Fixes: 68406d1085 (tests/unit: cleanups for test-io-channel-command)
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Suggested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   tests/unit/test-io-channel-command.c | 6 +++++-
>   1 file changed, 5 insertions(+), 1 deletion(-)

Reviewed-by: Thomas Huth <thuth@redhat.com>

[PATCH 05/12] gitlab: reduce default verbosity of cirrus run

[Alex Bennée]

We also truncate the echoing of the test log if we fail. Ideally we
would want the build aretefact to be available to gitlab but so far
how to do this eludes me.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Daniel P. Berrangé <berrange@redhat.com>
---
 .gitlab-ci.d/cirrus/build.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
index 7ef6af8d33..6563ff3c7a 100644
--- a/.gitlab-ci.d/cirrus/build.yml
+++ b/.gitlab-ci.d/cirrus/build.yml
@@ -32,6 +32,6 @@ build_task:
     - $MAKE -j$(sysctl -n hw.ncpu)
     - for TARGET in $TEST_TARGETS ;
       do
-        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1
-        || { cat meson-logs/testlog.txt; exit 1; } ;
+        $MAKE -j$(sysctl -n hw.ncpu) $TARGET
+        || { tail -n 200 meson-logs/testlog.txt; exit 1; } ;
       done
-- 
2.39.1

Re: [PATCH 05/12] gitlab: reduce default verbosity of cirrus run

[Thomas Huth]

On 15/02/2023 20.25, Alex Bennée wrote:
> We also truncate the echoing of the test log if we fail. Ideally we
> would want the build aretefact to be available to gitlab but so far
> how to do this eludes me.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   .gitlab-ci.d/cirrus/build.yml | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/.gitlab-ci.d/cirrus/build.yml b/.gitlab-ci.d/cirrus/build.yml
> index 7ef6af8d33..6563ff3c7a 100644
> --- a/.gitlab-ci.d/cirrus/build.yml
> +++ b/.gitlab-ci.d/cirrus/build.yml
> @@ -32,6 +32,6 @@ build_task:
>       - $MAKE -j$(sysctl -n hw.ncpu)
>       - for TARGET in $TEST_TARGETS ;
>         do
> -        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1
> -        || { cat meson-logs/testlog.txt; exit 1; } ;
> +        $MAKE -j$(sysctl -n hw.ncpu) $TARGET
> +        || { tail -n 200 meson-logs/testlog.txt; exit 1; } ;
>         done

I think it should be OK to publish the artifacts on cirrus-ci.com instead - 
you have to click a little bit more often, but you can still get the 
artifacts there, see:

  https://lore.kernel.org/qemu-devel/20230215142503.90660-1-thuth@redhat.com/

  Thomas

Re: [PATCH 05/12] gitlab: reduce default verbosity of cirrus run

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> On 15/02/2023 20.25, Alex Bennée wrote:
>> We also truncate the echoing of the test log if we fail. Ideally we
>> would want the build aretefact to be available to gitlab but so far
>> how to do this eludes me.
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Cc: Daniel P. Berrangé <berrange@redhat.com>
>> ---
>>   .gitlab-ci.d/cirrus/build.yml | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>> diff --git a/.gitlab-ci.d/cirrus/build.yml
>> b/.gitlab-ci.d/cirrus/build.yml
>> index 7ef6af8d33..6563ff3c7a 100644
>> --- a/.gitlab-ci.d/cirrus/build.yml
>> +++ b/.gitlab-ci.d/cirrus/build.yml
>> @@ -32,6 +32,6 @@ build_task:
>>       - $MAKE -j$(sysctl -n hw.ncpu)
>>       - for TARGET in $TEST_TARGETS ;
>>         do
>> -        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1
>> -        || { cat meson-logs/testlog.txt; exit 1; } ;
>> +        $MAKE -j$(sysctl -n hw.ncpu) $TARGET
>> +        || { tail -n 200 meson-logs/testlog.txt; exit 1; } ;
>>         done
>
> I think it should be OK to publish the artifacts on cirrus-ci.com
> instead - you have to click a little bit more often, but you can still
> get the artifacts there, see:
>
>  https://lore.kernel.org/qemu-devel/20230215142503.90660-1-thuth@redhat.com/

But dropping the V=1 also helps by reducing those chatty softfloat
tests. If we could merge that with yours. Are you sending a PR soon or
should I pull your patch into this series?

>
>  Thomas


-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 05/12] gitlab: reduce default verbosity of cirrus run

[Thomas Huth]

On 16/02/2023 09.02, Alex Bennée wrote:
> 
> Thomas Huth <thuth@redhat.com> writes:
> 
>> On 15/02/2023 20.25, Alex Bennée wrote:
>>> We also truncate the echoing of the test log if we fail. Ideally we
>>> would want the build aretefact to be available to gitlab but so far
>>> how to do this eludes me.
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> Cc: Daniel P. Berrangé <berrange@redhat.com>
>>> ---
>>>    .gitlab-ci.d/cirrus/build.yml | 4 ++--
>>>    1 file changed, 2 insertions(+), 2 deletions(-)
>>> diff --git a/.gitlab-ci.d/cirrus/build.yml
>>> b/.gitlab-ci.d/cirrus/build.yml
>>> index 7ef6af8d33..6563ff3c7a 100644
>>> --- a/.gitlab-ci.d/cirrus/build.yml
>>> +++ b/.gitlab-ci.d/cirrus/build.yml
>>> @@ -32,6 +32,6 @@ build_task:
>>>        - $MAKE -j$(sysctl -n hw.ncpu)
>>>        - for TARGET in $TEST_TARGETS ;
>>>          do
>>> -        $MAKE -j$(sysctl -n hw.ncpu) $TARGET V=1
>>> -        || { cat meson-logs/testlog.txt; exit 1; } ;
>>> +        $MAKE -j$(sysctl -n hw.ncpu) $TARGET
>>> +        || { tail -n 200 meson-logs/testlog.txt; exit 1; } ;
>>>          done
>>
>> I think it should be OK to publish the artifacts on cirrus-ci.com
>> instead - you have to click a little bit more often, but you can still
>> get the artifacts there, see:
>>
>>   https://lore.kernel.org/qemu-devel/20230215142503.90660-1-thuth@redhat.com/
> 
> But dropping the V=1 also helps by reducing those chatty softfloat
> tests. If we could merge that with yours. Are you sending a PR soon or
> should I pull your patch into this series?

I'm not planning a new pull request in the next few days, so feel free to 
pick my patch up or simply ignore it.

Anyway, the V=1 has just been added a little bit more than 2 years ago to 
address a different problem:

  https://gitlab.com/qemu-project/qemu/-/commit/2a5a79d1b57280edd

I'm fine with dropping the V=1 again, but it still feels like we're going 
around in circles here.

  Thomas

[PATCH 06/12] gitlab: extend custom runners with base_job_template

[Alex Bennée]

The base job template is responsible for controlling how we kick off
testing on our various branches. Rename and extend the
custom_runner_template so we can take advantage of all that control.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .gitlab-ci.d/custom-runners.yml                      |  3 ++-
 .gitlab-ci.d/custom-runners/ubuntu-20.04-s390x.yml   | 10 +++++-----
 .gitlab-ci.d/custom-runners/ubuntu-22.04-aarch32.yml |  2 +-
 .gitlab-ci.d/custom-runners/ubuntu-22.04-aarch64.yml | 10 +++++-----
 4 files changed, 13 insertions(+), 12 deletions(-)

diff --git a/.gitlab-ci.d/custom-runners.yml b/.gitlab-ci.d/custom-runners.yml
index 9fdc476c48..34a1e6f327 100644
--- a/.gitlab-ci.d/custom-runners.yml
+++ b/.gitlab-ci.d/custom-runners.yml
@@ -15,7 +15,8 @@ variables:
 
 # All custom runners can extend this template to upload the testlog
 # data as an artifact and also feed the junit report
-.custom_artifacts_template:
+.custom_runner_template:
+  extends: .base_job_template
   artifacts:
     name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
     expire_in: 7 days
diff --git a/.gitlab-ci.d/custom-runners/ubuntu-20.04-s390x.yml b/.gitlab-ci.d/custom-runners/ubuntu-20.04-s390x.yml
index f512eaeaa3..cdae6c5212 100644
--- a/.gitlab-ci.d/custom-runners/ubuntu-20.04-s390x.yml
+++ b/.gitlab-ci.d/custom-runners/ubuntu-20.04-s390x.yml
@@ -3,7 +3,7 @@
 # "Install basic packages to build QEMU on Ubuntu 20.04/20.04"
 
 ubuntu-20.04-s390x-all-linux-static:
- extends: .custom_artifacts_template
+ extends: .custom_runner_template
  needs: []
  stage: build
  tags:
@@ -24,7 +24,7 @@ ubuntu-20.04-s390x-all-linux-static:
  - make --output-sync -j`nproc` check
 
 ubuntu-20.04-s390x-all:
- extends: .custom_artifacts_template
+ extends: .custom_runner_template
  needs: []
  stage: build
  tags:
@@ -43,7 +43,7 @@ ubuntu-20.04-s390x-all:
  - make --output-sync -j`nproc` check
 
 ubuntu-20.04-s390x-alldbg:
- extends: .custom_artifacts_template
+ extends: .custom_runner_template
  needs: []
  stage: build
  tags:
@@ -66,7 +66,7 @@ ubuntu-20.04-s390x-alldbg:
  - make --output-sync -j`nproc` check
 
 ubuntu-20.04-s390x-clang:
- extends: .custom_artifacts_template
+ extends: .custom_runner_template
  needs: []
  stage: build
  tags:
@@ -108,7 +108,7 @@ ubuntu-20.04-s390x-tci:
  - make --output-sync -j`nproc`
 
 ubuntu-20.04-s390x-notcg:
- extends: .custom_artifacts_template
+ extends: .custom_runner_template
  needs: []
  stage: build
  tags:
diff --git a/.gitlab-ci.d/custom-runners/ubuntu-22.04-aarch32.yml b/.gitlab-ci.d/custom-runners/ubuntu-22.04-aarch32.yml
index 42137aaf2a..50e5646a44 100644
--- a/.gitlab-ci.d/custom-runners/ubuntu-22.04-aarch32.yml
+++ b/.gitlab-ci.d/custom-runners/ubuntu-22.04-aarch32.yml
@@ -3,7 +3,7 @@
 # "Install basic packages to build QEMU on Ubuntu 20.04"
 
 ubuntu-22.04-aarch32-all:
- extends: .custom_artifacts_template
+ extends: .custom_runner_template
  needs: []
  stage: build
  tags:
diff --git a/.gitlab-ci.d/custom-runners/ubuntu-22.04-aarch64.yml b/.gitlab-ci.d/custom-runners/ubuntu-22.04-aarch64.yml
index 8ba85be440..13e14a0f87 100644
--- a/.gitlab-ci.d/custom-runners/ubuntu-22.04-aarch64.yml
+++ b/.gitlab-ci.d/custom-runners/ubuntu-22.04-aarch64.yml
@@ -3,7 +3,7 @@
 # "Install basic packages to build QEMU on Ubuntu 20.04"
 
 ubuntu-22.04-aarch64-all-linux-static:
- extends: .custom_artifacts_template
+ extends: .custom_runner_template
  needs: []
  stage: build
  tags:
@@ -24,7 +24,7 @@ ubuntu-22.04-aarch64-all-linux-static:
  - make --output-sync -j`nproc --ignore=40` check
 
 ubuntu-22.04-aarch64-all:
- extends: .custom_artifacts_template
+ extends: .custom_runner_template
  needs: []
  stage: build
  tags:
@@ -46,7 +46,7 @@ ubuntu-22.04-aarch64-all:
  - make --output-sync -j`nproc --ignore=40` check
 
 ubuntu-22.04-aarch64-alldbg:
- extends: .custom_artifacts_template
+ extends: .custom_runner_template
  needs: []
  stage: build
  tags:
@@ -65,7 +65,7 @@ ubuntu-22.04-aarch64-alldbg:
  - make --output-sync -j`nproc --ignore=40` check
 
 ubuntu-22.04-aarch64-clang:
- extends: .custom_artifacts_template
+ extends: .custom_runner_template
  needs: []
  stage: build
  tags:
@@ -107,7 +107,7 @@ ubuntu-22.04-aarch64-tci:
  - make --output-sync -j`nproc --ignore=40`
 
 ubuntu-22.04-aarch64-notcg:
- extends: .custom_artifacts_template
+ extends: .custom_runner_template
  needs: []
  stage: build
  tags:
-- 
2.39.1

Re: [PATCH 06/12] gitlab: extend custom runners with base_job_template

[Thomas Huth]

On 15/02/2023 20.25, Alex Bennée wrote:
> The base job template is responsible for controlling how we kick off
> testing on our various branches. Rename and extend the
> custom_runner_template so we can take advantage of all that control.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   .gitlab-ci.d/custom-runners.yml                      |  3 ++-
>   .gitlab-ci.d/custom-runners/ubuntu-20.04-s390x.yml   | 10 +++++-----
>   .gitlab-ci.d/custom-runners/ubuntu-22.04-aarch32.yml |  2 +-
>   .gitlab-ci.d/custom-runners/ubuntu-22.04-aarch64.yml | 10 +++++-----
>   4 files changed, 13 insertions(+), 12 deletions(-)

Reviewed-by: Thomas Huth <thuth@redhat.com>

[PATCH 07/12] testing: update ubuntu2004 to ubuntu2204

[Alex Bennée]

The 22.04 LTS release has been out for almost a year now so its time
to update all the remaining images to the current LTS. We can also
drop some hacks we need for older clang TSAN support.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 docs/devel/testing.rst                        |  4 ++--
 .gitlab-ci.d/buildtest.yml                    | 22 +++++++++----------
 .gitlab-ci.d/containers.yml                   |  4 ++--
 .../{ubuntu2004.docker => ubuntu2204.docker}  | 16 +++++---------
 tests/docker/test-tsan                        |  2 +-
 tests/lcitool/refresh                         | 10 +--------
 6 files changed, 23 insertions(+), 35 deletions(-)
 rename tests/docker/dockerfiles/{ubuntu2004.docker => ubuntu2204.docker} (91%)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index e10c47b5a7..309a575abe 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -574,13 +574,13 @@ https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
 
 Thread Sanitizer in Docker
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
-TSan is currently supported in the ubuntu2004 docker.
+TSan is currently supported in the ubuntu2204 docker.
 
 The test-tsan test will build using TSan and then run make check.
 
 .. code::
 
-  make docker-test-tsan@ubuntu2004
+  make docker-test-tsan@ubuntu2204
 
 TSan warnings under docker are placed in files located at build/tsan/.
 
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 8f332fc36f..be319ab2d0 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -38,9 +38,9 @@ avocado-system-alpine:
 build-system-ubuntu:
   extends: .native_build_job_template
   needs:
-    job: amd64-ubuntu2004-container
+    job: amd64-ubuntu2204-container
   variables:
-    IMAGE: ubuntu2004
+    IMAGE: ubuntu2204
     CONFIGURE_ARGS: --enable-docs --enable-fdt=system --enable-capstone
     TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
       microblazeel-softmmu mips64el-softmmu
@@ -56,7 +56,7 @@ check-system-ubuntu:
     - job: build-system-ubuntu
       artifacts: true
   variables:
-    IMAGE: ubuntu2004
+    IMAGE: ubuntu2204
     MAKE_CHECK_ARGS: check
 
 avocado-system-ubuntu:
@@ -65,7 +65,7 @@ avocado-system-ubuntu:
     - job: build-system-ubuntu
       artifacts: true
   variables:
-    IMAGE: ubuntu2004
+    IMAGE: ubuntu2204
     MAKE_CHECK_ARGS: check-avocado
 
 build-system-debian:
@@ -459,10 +459,10 @@ avocado-cfi-x86_64:
 tsan-build:
   extends: .native_build_job_template
   needs:
-    job: amd64-ubuntu2004-container
+    job: amd64-ubuntu2204-container
   variables:
-    IMAGE: ubuntu2004
-    CONFIGURE_ARGS: --enable-tsan --cc=clang-10 --cxx=clang++-10
+    IMAGE: ubuntu2204
+    CONFIGURE_ARGS: --enable-tsan --cc=clang --cxx=clang++
           --enable-trace-backends=ust --enable-fdt=system --disable-slirp
     TARGETS: x86_64-softmmu ppc64-softmmu riscv64-softmmu x86_64-linux-user
     MAKE_CHECK_ARGS: bench V=1
@@ -471,10 +471,10 @@ tsan-build:
 gcov:
   extends: .native_build_job_template
   needs:
-    job: amd64-ubuntu2004-container
+    job: amd64-ubuntu2204-container
   timeout: 80m
   variables:
-    IMAGE: ubuntu2004
+    IMAGE: ubuntu2204
     CONFIGURE_ARGS: --enable-gcov
     TARGETS: aarch64-softmmu ppc64-softmmu s390x-softmmu x86_64-softmmu
     MAKE_CHECK_ARGS: check
@@ -539,9 +539,9 @@ build-tci:
 build-coroutine-sigaltstack:
   extends: .native_build_job_template
   needs:
-    job: amd64-ubuntu2004-container
+    job: amd64-ubuntu2204-container
   variables:
-    IMAGE: ubuntu2004
+    IMAGE: ubuntu2204
     CONFIGURE_ARGS: --with-coroutine=sigaltstack --disable-tcg
                     --enable-trace-backends=ftrace
     MAKE_CHECK_ARGS: check-unit
diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index 96d2a3b58b..8637a13d86 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -13,10 +13,10 @@ amd64-debian-container:
   variables:
     NAME: debian-amd64
 
-amd64-ubuntu2004-container:
+amd64-ubuntu2204-container:
   extends: .container_job_template
   variables:
-    NAME: ubuntu2004
+    NAME: ubuntu2204
 
 amd64-opensuse-leap-container:
   extends: .container_job_template
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2204.docker
similarity index 91%
rename from tests/docker/dockerfiles/ubuntu2004.docker
rename to tests/docker/dockerfiles/ubuntu2204.docker
index f34d88d33d..30b9e56793 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2204.docker
@@ -1,10 +1,10 @@
 # THIS FILE WAS AUTO-GENERATED
 #
-#  $ lcitool dockerfile --layers all ubuntu-2004 qemu
+#  $ lcitool dockerfile --layers all ubuntu-2204 qemu
 #
 # https://gitlab.com/libvirt/libvirt-ci
 
-FROM docker.io/library/ubuntu:20.04
+FROM docker.io/library/ubuntu:22.04
 
 RUN export DEBIAN_FRONTEND=noninteractive && \
     apt-get update && \
@@ -14,7 +14,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       bash \
                       bc \
                       bison \
-                      bsdmainutils \
+                      bsdextrautils \
                       bzip2 \
                       ca-certificates \
                       ccache \
@@ -36,6 +36,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       libasan5 \
                       libasound2-dev \
                       libattr1-dev \
+                      libbpf-dev \
                       libbrlapi-dev \
                       libbz2-dev \
                       libc6-dev \
@@ -90,6 +91,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       libtasn1-6-dev \
                       libubsan1 \
                       libudev-dev \
+                      liburing-dev \
                       libusb-1.0-0-dev \
                       libusbredirhost-dev \
                       libvdeplug-dev \
@@ -100,6 +102,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       llvm \
                       locales \
                       make \
+                      meson \
                       multipath-tools \
                       ncat \
                       nettle-dev \
@@ -111,11 +114,9 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       python3-opencv \
                       python3-pillow \
                       python3-pip \
-                      python3-setuptools \
                       python3-sphinx \
                       python3-sphinx-rtd-theme \
                       python3-venv \
-                      python3-wheel \
                       python3-yaml \
                       rpm2cpio \
                       sed \
@@ -139,13 +140,8 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
     ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
 
-RUN /usr/bin/pip3 install meson==0.56.0
-
 ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
 ENV LANG "en_US.UTF-8"
 ENV MAKE "/usr/bin/make"
 ENV NINJA "/usr/bin/ninja"
 ENV PYTHON "/usr/bin/python3"
-# Apply patch https://reviews.llvm.org/D75820
-# This is required for TSan in clang-10 to compile with QEMU.
-RUN sed -i 's/^const/static const/g' /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h
diff --git a/tests/docker/test-tsan b/tests/docker/test-tsan
index 53d90d2f79..f6d6590e39 100755
--- a/tests/docker/test-tsan
+++ b/tests/docker/test-tsan
@@ -21,7 +21,7 @@ setup_tsan()
     tsan_log_dir="/tmp/qemu-test/build/tsan"
     mkdir -p $tsan_log_dir > /dev/null || true
     EXTRA_CONFIGURE_OPTS="${EXTRA_CONFIGURE_OPTS} --enable-tsan \
-                          --cc=clang-10 --cxx=clang++-10 \
+                          --cc=clang --cxx=clang++ \
                           --disable-werror --extra-cflags=-O0"
     # detect deadlocks is false currently simply because
     # TSan crashes immediately with deadlock detector enabled.
diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index a5ea0efc3b..eeee9a7b34 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -69,13 +69,6 @@ def generate_cirrus(target, trailer=None):
     generate(filename, cmd, trailer)
 
 
-ubuntu2004_tsanhack = [
-    "# Apply patch https://reviews.llvm.org/D75820\n",
-    "# This is required for TSan in clang-10 to compile with QEMU.\n",
-    "RUN sed -i 's/^const/static const/g' /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h\n"
-]
-
-
 # Netmap still needs to be manually built as it is yet to be packaged
 # into a distro. We also add cscope and gtags which are used in the CI
 # test
@@ -113,8 +106,7 @@ try:
                         trailer="".join(debian11_extras))
     generate_dockerfile("fedora", "fedora-37")
     generate_dockerfile("opensuse-leap", "opensuse-leap-153")
-    generate_dockerfile("ubuntu2004", "ubuntu-2004",
-                        trailer="".join(ubuntu2004_tsanhack))
+    generate_dockerfile("ubuntu2204", "ubuntu-2204")
 
     #
     # Cross compiling builds
-- 
2.39.1

Re: [PATCH 07/12] testing: update ubuntu2004 to ubuntu2204

[Richard Henderson]

On 2/15/23 09:25, Alex Bennée wrote:
> The 22.04 LTS release has been out for almost a year now so its time
> to update all the remaining images to the current LTS. We can also
> drop some hacks we need for older clang TSAN support.
> 
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
> ---
>   docs/devel/testing.rst                        |  4 ++--
>   .gitlab-ci.d/buildtest.yml                    | 22 +++++++++----------
>   .gitlab-ci.d/containers.yml                   |  4 ++--
>   .../{ubuntu2004.docker => ubuntu2204.docker}  | 16 +++++---------
>   tests/docker/test-tsan                        |  2 +-
>   tests/lcitool/refresh                         | 10 +--------
>   6 files changed, 23 insertions(+), 35 deletions(-)
>   rename tests/docker/dockerfiles/{ubuntu2004.docker => ubuntu2204.docker} (91%)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [PATCH 07/12] testing: update ubuntu2004 to ubuntu2204

[Thomas Huth]

On 15/02/2023 20.25, Alex Bennée wrote:
> The 22.04 LTS release has been out for almost a year now so its time
> to update all the remaining images to the current LTS. We can also
> drop some hacks we need for older clang TSAN support.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   docs/devel/testing.rst                        |  4 ++--
>   .gitlab-ci.d/buildtest.yml                    | 22 +++++++++----------
>   .gitlab-ci.d/containers.yml                   |  4 ++--
>   .../{ubuntu2004.docker => ubuntu2204.docker}  | 16 +++++---------
>   tests/docker/test-tsan                        |  2 +-
>   tests/lcitool/refresh                         | 10 +--------
>   6 files changed, 23 insertions(+), 35 deletions(-)
>   rename tests/docker/dockerfiles/{ubuntu2004.docker => ubuntu2204.docker} (91%)

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 07/12] testing: update ubuntu2004 to ubuntu2204

[John Snow]

On Wed, Feb 15, 2023 at 2:25 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> The 22.04 LTS release has been out for almost a year now so its time
> to update all the remaining images to the current LTS. We can also
> drop some hacks we need for older clang TSAN support.

We still support Ubuntu 20.04 until 2024 though, don't we? Is it safe
to not test this platform?

I've long been uncertain about what our policy actually is for docker
tests, if we want to test every platform we support or only some of
them; and if it's only some of them, when do we choose the older and
when do we choose the newer?



>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  docs/devel/testing.rst                        |  4 ++--
>  .gitlab-ci.d/buildtest.yml                    | 22 +++++++++----------
>  .gitlab-ci.d/containers.yml                   |  4 ++--
>  .../{ubuntu2004.docker => ubuntu2204.docker}  | 16 +++++---------
>  tests/docker/test-tsan                        |  2 +-
>  tests/lcitool/refresh                         | 10 +--------
>  6 files changed, 23 insertions(+), 35 deletions(-)
>  rename tests/docker/dockerfiles/{ubuntu2004.docker => ubuntu2204.docker} (91%)
>
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index e10c47b5a7..309a575abe 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -574,13 +574,13 @@ https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
>
>  Thread Sanitizer in Docker
>  ~~~~~~~~~~~~~~~~~~~~~~~~~~
> -TSan is currently supported in the ubuntu2004 docker.
> +TSan is currently supported in the ubuntu2204 docker.
>
>  The test-tsan test will build using TSan and then run make check.
>
>  .. code::
>
> -  make docker-test-tsan@ubuntu2004
> +  make docker-test-tsan@ubuntu2204
>
>  TSan warnings under docker are placed in files located at build/tsan/.
>
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 8f332fc36f..be319ab2d0 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -38,9 +38,9 @@ avocado-system-alpine:
>  build-system-ubuntu:
>    extends: .native_build_job_template
>    needs:
> -    job: amd64-ubuntu2004-container
> +    job: amd64-ubuntu2204-container
>    variables:
> -    IMAGE: ubuntu2004
> +    IMAGE: ubuntu2204
>      CONFIGURE_ARGS: --enable-docs --enable-fdt=system --enable-capstone
>      TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
>        microblazeel-softmmu mips64el-softmmu
> @@ -56,7 +56,7 @@ check-system-ubuntu:
>      - job: build-system-ubuntu
>        artifacts: true
>    variables:
> -    IMAGE: ubuntu2004
> +    IMAGE: ubuntu2204
>      MAKE_CHECK_ARGS: check
>
>  avocado-system-ubuntu:
> @@ -65,7 +65,7 @@ avocado-system-ubuntu:
>      - job: build-system-ubuntu
>        artifacts: true
>    variables:
> -    IMAGE: ubuntu2004
> +    IMAGE: ubuntu2204
>      MAKE_CHECK_ARGS: check-avocado
>
>  build-system-debian:
> @@ -459,10 +459,10 @@ avocado-cfi-x86_64:
>  tsan-build:
>    extends: .native_build_job_template
>    needs:
> -    job: amd64-ubuntu2004-container
> +    job: amd64-ubuntu2204-container
>    variables:
> -    IMAGE: ubuntu2004
> -    CONFIGURE_ARGS: --enable-tsan --cc=clang-10 --cxx=clang++-10
> +    IMAGE: ubuntu2204
> +    CONFIGURE_ARGS: --enable-tsan --cc=clang --cxx=clang++
>            --enable-trace-backends=ust --enable-fdt=system --disable-slirp
>      TARGETS: x86_64-softmmu ppc64-softmmu riscv64-softmmu x86_64-linux-user
>      MAKE_CHECK_ARGS: bench V=1
> @@ -471,10 +471,10 @@ tsan-build:
>  gcov:
>    extends: .native_build_job_template
>    needs:
> -    job: amd64-ubuntu2004-container
> +    job: amd64-ubuntu2204-container
>    timeout: 80m
>    variables:
> -    IMAGE: ubuntu2004
> +    IMAGE: ubuntu2204
>      CONFIGURE_ARGS: --enable-gcov
>      TARGETS: aarch64-softmmu ppc64-softmmu s390x-softmmu x86_64-softmmu
>      MAKE_CHECK_ARGS: check
> @@ -539,9 +539,9 @@ build-tci:
>  build-coroutine-sigaltstack:
>    extends: .native_build_job_template
>    needs:
> -    job: amd64-ubuntu2004-container
> +    job: amd64-ubuntu2204-container
>    variables:
> -    IMAGE: ubuntu2004
> +    IMAGE: ubuntu2204
>      CONFIGURE_ARGS: --with-coroutine=sigaltstack --disable-tcg
>                      --enable-trace-backends=ftrace
>      MAKE_CHECK_ARGS: check-unit
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> index 96d2a3b58b..8637a13d86 100644
> --- a/.gitlab-ci.d/containers.yml
> +++ b/.gitlab-ci.d/containers.yml
> @@ -13,10 +13,10 @@ amd64-debian-container:
>    variables:
>      NAME: debian-amd64
>
> -amd64-ubuntu2004-container:
> +amd64-ubuntu2204-container:
>    extends: .container_job_template
>    variables:
> -    NAME: ubuntu2004
> +    NAME: ubuntu2204
>
>  amd64-opensuse-leap-container:
>    extends: .container_job_template
> diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2204.docker
> similarity index 91%
> rename from tests/docker/dockerfiles/ubuntu2004.docker
> rename to tests/docker/dockerfiles/ubuntu2204.docker
> index f34d88d33d..30b9e56793 100644
> --- a/tests/docker/dockerfiles/ubuntu2004.docker
> +++ b/tests/docker/dockerfiles/ubuntu2204.docker
> @@ -1,10 +1,10 @@
>  # THIS FILE WAS AUTO-GENERATED
>  #
> -#  $ lcitool dockerfile --layers all ubuntu-2004 qemu
> +#  $ lcitool dockerfile --layers all ubuntu-2204 qemu
>  #
>  # https://gitlab.com/libvirt/libvirt-ci
>
> -FROM docker.io/library/ubuntu:20.04
> +FROM docker.io/library/ubuntu:22.04
>
>  RUN export DEBIAN_FRONTEND=noninteractive && \
>      apt-get update && \
> @@ -14,7 +14,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>                        bash \
>                        bc \
>                        bison \
> -                      bsdmainutils \
> +                      bsdextrautils \
>                        bzip2 \
>                        ca-certificates \
>                        ccache \
> @@ -36,6 +36,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>                        libasan5 \
>                        libasound2-dev \
>                        libattr1-dev \
> +                      libbpf-dev \
>                        libbrlapi-dev \
>                        libbz2-dev \
>                        libc6-dev \
> @@ -90,6 +91,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>                        libtasn1-6-dev \
>                        libubsan1 \
>                        libudev-dev \
> +                      liburing-dev \
>                        libusb-1.0-0-dev \
>                        libusbredirhost-dev \
>                        libvdeplug-dev \
> @@ -100,6 +102,7 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>                        llvm \
>                        locales \
>                        make \
> +                      meson \
>                        multipath-tools \
>                        ncat \
>                        nettle-dev \
> @@ -111,11 +114,9 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>                        python3-opencv \
>                        python3-pillow \
>                        python3-pip \
> -                      python3-setuptools \
>                        python3-sphinx \
>                        python3-sphinx-rtd-theme \
>                        python3-venv \
> -                      python3-wheel \
>                        python3-yaml \
>                        rpm2cpio \
>                        sed \
> @@ -139,13 +140,8 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>      ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
>      ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
>
> -RUN /usr/bin/pip3 install meson==0.56.0
> -
>  ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
>  ENV LANG "en_US.UTF-8"
>  ENV MAKE "/usr/bin/make"
>  ENV NINJA "/usr/bin/ninja"
>  ENV PYTHON "/usr/bin/python3"
> -# Apply patch https://reviews.llvm.org/D75820
> -# This is required for TSan in clang-10 to compile with QEMU.
> -RUN sed -i 's/^const/static const/g' /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h
> diff --git a/tests/docker/test-tsan b/tests/docker/test-tsan
> index 53d90d2f79..f6d6590e39 100755
> --- a/tests/docker/test-tsan
> +++ b/tests/docker/test-tsan
> @@ -21,7 +21,7 @@ setup_tsan()
>      tsan_log_dir="/tmp/qemu-test/build/tsan"
>      mkdir -p $tsan_log_dir > /dev/null || true
>      EXTRA_CONFIGURE_OPTS="${EXTRA_CONFIGURE_OPTS} --enable-tsan \
> -                          --cc=clang-10 --cxx=clang++-10 \
> +                          --cc=clang --cxx=clang++ \
>                            --disable-werror --extra-cflags=-O0"
>      # detect deadlocks is false currently simply because
>      # TSan crashes immediately with deadlock detector enabled.
> diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
> index a5ea0efc3b..eeee9a7b34 100755
> --- a/tests/lcitool/refresh
> +++ b/tests/lcitool/refresh
> @@ -69,13 +69,6 @@ def generate_cirrus(target, trailer=None):
>      generate(filename, cmd, trailer)
>
>
> -ubuntu2004_tsanhack = [
> -    "# Apply patch https://reviews.llvm.org/D75820\n",
> -    "# This is required for TSan in clang-10 to compile with QEMU.\n",
> -    "RUN sed -i 's/^const/static const/g' /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h\n"
> -]
> -
> -
>  # Netmap still needs to be manually built as it is yet to be packaged
>  # into a distro. We also add cscope and gtags which are used in the CI
>  # test
> @@ -113,8 +106,7 @@ try:
>                          trailer="".join(debian11_extras))
>      generate_dockerfile("fedora", "fedora-37")
>      generate_dockerfile("opensuse-leap", "opensuse-leap-153")
> -    generate_dockerfile("ubuntu2004", "ubuntu-2004",
> -                        trailer="".join(ubuntu2004_tsanhack))
> +    generate_dockerfile("ubuntu2204", "ubuntu-2204")
>
>      #
>      # Cross compiling builds
> --
> 2.39.1
>

Re: [PATCH 07/12] testing: update ubuntu2004 to ubuntu2204

[Daniel P. Berrangé]

On Thu, Feb 16, 2023 at 01:15:30PM -0500, John Snow wrote:
> On Wed, Feb 15, 2023 at 2:25 PM Alex Bennée <alex.bennee@linaro.org> wrote:
> >
> > The 22.04 LTS release has been out for almost a year now so its time
> > to update all the remaining images to the current LTS. We can also
> > drop some hacks we need for older clang TSAN support.
> 
> We still support Ubuntu 20.04 until 2024 though, don't we? Is it safe
> to not test this platform?
> 
> I've long been uncertain about what our policy actually is for docker
> tests, if we want to test every platform we support or only some of
> them; and if it's only some of them, when do we choose the older and
> when do we choose the newer?

Ideally we would test both the oldest & newest versions of each
distro we support. Practically though, we're compromised by the
limited CI resources available.

Dropping older Ubuntu images is a reasonable tradeoff, since we
still have Debian images covered in CI. Debian can be thought
of as an older version of Ubuntu to some extent, giving coverage
that will mitigate the risks of dropping 20.04.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 07/12] testing: update ubuntu2004 to ubuntu2204

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2582 bytes --]

On Thu, Feb 16, 2023, 2:44 PM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Thu, Feb 16, 2023 at 01:15:30PM -0500, John Snow wrote:
> > On Wed, Feb 15, 2023 at 2:25 PM Alex Bennée <alex.bennee@linaro.org>
> wrote:
> > >
> > > The 22.04 LTS release has been out for almost a year now so its time
> > > to update all the remaining images to the current LTS. We can also
> > > drop some hacks we need for older clang TSAN support.
> >
> > We still support Ubuntu 20.04 until 2024 though, don't we? Is it safe
> > to not test this platform?
> >
> > I've long been uncertain about what our policy actually is for docker
> > tests, if we want to test every platform we support or only some of
> > them; and if it's only some of them, when do we choose the older and
> > when do we choose the newer?
>
> Ideally we would test both the oldest & newest versions of each
> distro we support. Practically though, we're compromised by the
> limited CI resources available.
>

Yes, understood.


> Dropping older Ubuntu images is a reasonable tradeoff, since we
> still have Debian images covered in CI. Debian can be thought
> of as an older version of Ubuntu to some extent, giving coverage
> that will mitigate the risks of dropping 20.04.
>

Okay, I'll take your word for that. I am not personally familiar with how
much those distros diverge; I know Ubuntu is debian-based but that's the
extent of my knowledge as I don't daily-drive either.

So, firstly:

Reviewed-by: John Snow <jsnow@redhat.com>

because I suspect we all have our reasons and I also agree testing newer is
generally of higher value than testing older.

However, would it be possible to keep the older Ubuntu test as a manual
execution that we could invoke at will, only during RC testing phase? If
it's not a lot of work, I could even check that in myself as a follow-up if
it isn't unwanted.

I find that "oldest version of x" is quite useful to me for testing Python
stuff in particular, as that ecosystem moves pretty fast. It'd be mighty
convenient to me in particular to keep an old Ubuntu test around to run
manually as needed.

(Heck, even if it wasn't on CI at all but was just a container I could run
locally, that would still be quite useful.)

Whaddaya think?


> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>

[-- Attachment #2: Type: text/html, Size: 4617 bytes --]

Re: [PATCH 07/12] testing: update ubuntu2004 to ubuntu2204

[Daniel P. Berrangé]

On Fri, Feb 17, 2023 at 11:35:44AM -0500, John Snow wrote:
> On Thu, Feb 16, 2023, 2:44 PM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> 
> > On Thu, Feb 16, 2023 at 01:15:30PM -0500, John Snow wrote:
> > > On Wed, Feb 15, 2023 at 2:25 PM Alex Bennée <alex.bennee@linaro.org>
> > wrote:
> > > >
> > > > The 22.04 LTS release has been out for almost a year now so its time
> > > > to update all the remaining images to the current LTS. We can also
> > > > drop some hacks we need for older clang TSAN support.
> > >
> > > We still support Ubuntu 20.04 until 2024 though, don't we? Is it safe
> > > to not test this platform?
> > >
> > > I've long been uncertain about what our policy actually is for docker
> > > tests, if we want to test every platform we support or only some of
> > > them; and if it's only some of them, when do we choose the older and
> > > when do we choose the newer?
> >
> > Ideally we would test both the oldest & newest versions of each
> > distro we support. Practically though, we're compromised by the
> > limited CI resources available.
> >
> 
> Yes, understood.
> 
> 
> > Dropping older Ubuntu images is a reasonable tradeoff, since we
> > still have Debian images covered in CI. Debian can be thought
> > of as an older version of Ubuntu to some extent, giving coverage
> > that will mitigate the risks of dropping 20.04.
> >
> 
> Okay, I'll take your word for that. I am not personally familiar with how
> much those distros diverge; I know Ubuntu is debian-based but that's the
> extent of my knowledge as I don't daily-drive either.
> 
> So, firstly:
> 
> Reviewed-by: John Snow <jsnow@redhat.com>
> 
> because I suspect we all have our reasons and I also agree testing newer is
> generally of higher value than testing older.
> 
> However, would it be possible to keep the older Ubuntu test as a manual
> execution that we could invoke at will, only during RC testing phase? If
> it's not a lot of work, I could even check that in myself as a follow-up if
> it isn't unwanted.
> 
> I find that "oldest version of x" is quite useful to me for testing Python
> stuff in particular, as that ecosystem moves pretty fast. It'd be mighty
> convenient to me in particular to keep an old Ubuntu test around to run
> manually as needed.
> 
> (Heck, even if it wasn't on CI at all but was just a container I could run
> locally, that would still be quite useful.)
> 
> Whaddaya think?

It would be pretty trivial to have tests/docker/dockerfiles contain
Dockerfiles for *every* supported distro version we have, and then
only build & test a subset in CI. It would merely suggest that we
change our naming convention so the dockerfiles in that dir include
the version. Basically adopting the standard libvirt-ci naming
convention for targets of $OSNAME-$OSVERSION:

$ lcitool targets
almalinux-8
almalinux-9
alpine-315
alpine-316
alpine-edge
centos-stream-8
centos-stream-9
debian-10
debian-11
debian-sid
fedora-36
fedora-37
fedora-rawhide
freebsd-12
freebsd-13
freebsd-current
macos-12
macos-13
opensuse-leap-153
opensuse-leap-154
opensuse-tumbleweed
ubuntu-1804
ubuntu-2004
ubuntu-2204

Contributors can then use 'make docker-XXXX' to run build tests
locally on specific distros when they need to test something
that isn't covered by default in out gating CI


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 07/12] testing: update ubuntu2004 to ubuntu2204

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 4414 bytes --]

On Fri, Feb 17, 2023, 12:14 PM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Fri, Feb 17, 2023 at 11:35:44AM -0500, John Snow wrote:
> > On Thu, Feb 16, 2023, 2:44 PM Daniel P. Berrangé <berrange@redhat.com>
> > wrote:
> >
> > > On Thu, Feb 16, 2023 at 01:15:30PM -0500, John Snow wrote:
> > > > On Wed, Feb 15, 2023 at 2:25 PM Alex Bennée <alex.bennee@linaro.org>
> > > wrote:
> > > > >
> > > > > The 22.04 LTS release has been out for almost a year now so its
> time
> > > > > to update all the remaining images to the current LTS. We can also
> > > > > drop some hacks we need for older clang TSAN support.
> > > >
> > > > We still support Ubuntu 20.04 until 2024 though, don't we? Is it safe
> > > > to not test this platform?
> > > >
> > > > I've long been uncertain about what our policy actually is for docker
> > > > tests, if we want to test every platform we support or only some of
> > > > them; and if it's only some of them, when do we choose the older and
> > > > when do we choose the newer?
> > >
> > > Ideally we would test both the oldest & newest versions of each
> > > distro we support. Practically though, we're compromised by the
> > > limited CI resources available.
> > >
> >
> > Yes, understood.
> >
> >
> > > Dropping older Ubuntu images is a reasonable tradeoff, since we
> > > still have Debian images covered in CI. Debian can be thought
> > > of as an older version of Ubuntu to some extent, giving coverage
> > > that will mitigate the risks of dropping 20.04.
> > >
> >
> > Okay, I'll take your word for that. I am not personally familiar with how
> > much those distros diverge; I know Ubuntu is debian-based but that's the
> > extent of my knowledge as I don't daily-drive either.
> >
> > So, firstly:
> >
> > Reviewed-by: John Snow <jsnow@redhat.com>
> >
> > because I suspect we all have our reasons and I also agree testing newer
> is
> > generally of higher value than testing older.
> >
> > However, would it be possible to keep the older Ubuntu test as a manual
> > execution that we could invoke at will, only during RC testing phase? If
> > it's not a lot of work, I could even check that in myself as a follow-up
> if
> > it isn't unwanted.
> >
> > I find that "oldest version of x" is quite useful to me for testing
> Python
> > stuff in particular, as that ecosystem moves pretty fast. It'd be mighty
> > convenient to me in particular to keep an old Ubuntu test around to run
> > manually as needed.
> >
> > (Heck, even if it wasn't on CI at all but was just a container I could
> run
> > locally, that would still be quite useful.)
> >
> > Whaddaya think?
>
> It would be pretty trivial to have tests/docker/dockerfiles contain
> Dockerfiles for *every* supported distro version we have, and then
> only build & test a subset in CI. It would merely suggest that we
> change our naming convention so the dockerfiles in that dir include
> the version. Basically adopting the standard libvirt-ci naming
> convention for targets of $OSNAME-$OSVERSION:
>
> $ lcitool targets
> almalinux-8
> almalinux-9
> alpine-315
> alpine-316
> alpine-edge
> centos-stream-8
> centos-stream-9
> debian-10
> debian-11
> debian-sid
> fedora-36
> fedora-37
> fedora-rawhide
> freebsd-12
> freebsd-13
> freebsd-current
> macos-12
> macos-13
>

Wait, what? How does this work??

opensuse-leap-153
> opensuse-leap-154
> opensuse-tumbleweed
> ubuntu-1804
> ubuntu-2004
> ubuntu-2204
>
> Contributors can then use 'make docker-XXXX' to run build tests
> locally on specific distros when they need to test something
> that isn't covered by default in out gating CI
>

OK, I might follow up on this, then. I would find this useful for proving
certain python build system changes are not disruptive.

In contrast to C world, I find modern Pythonisms sneak in with quite an
increased frequency, so having manual testing for the oldest platforms has
some value there, but only every once in a while. Not worth our CI minutes.

Carry on as normal for this series, please and thank you!


>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>

[-- Attachment #2: Type: text/html, Size: 6618 bytes --]

Re: [PATCH 07/12] testing: update ubuntu2004 to ubuntu2204

[Philippe Mathieu-Daudé]

On 17/2/23 18:14, Daniel P. Berrangé wrote:
> On Fri, Feb 17, 2023 at 11:35:44AM -0500, John Snow wrote:

>> However, would it be possible to keep the older Ubuntu test as a manual
>> execution that we could invoke at will, only during RC testing phase? If
>> it's not a lot of work, I could even check that in myself as a follow-up if
>> it isn't unwanted.
>>
>> I find that "oldest version of x" is quite useful to me for testing Python
>> stuff in particular, as that ecosystem moves pretty fast. It'd be mighty
>> convenient to me in particular to keep an old Ubuntu test around to run
>> manually as needed.
>>
>> (Heck, even if it wasn't on CI at all but was just a container I could run
>> locally, that would still be quite useful.)
>>
>> Whaddaya think?
> 
> It would be pretty trivial to have tests/docker/dockerfiles contain
> Dockerfiles for *every* supported distro version we have, and then
> only build & test a subset in CI. It would merely suggest that we

That would be great! Could be added as byte-sized issue.

> change our naming convention so the dockerfiles in that dir include
> the version. Basically adopting the standard libvirt-ci naming
> convention for targets of $OSNAME-$OSVERSION:
> 
> $ lcitool targets
> almalinux-8
> almalinux-9
> alpine-315
> alpine-316
> alpine-edge
> centos-stream-8
> centos-stream-9
> debian-10
> debian-11
> debian-sid
> fedora-36
> fedora-37
> fedora-rawhide
> freebsd-12
> freebsd-13
> freebsd-current
> macos-12
> macos-13
> opensuse-leap-153
> opensuse-leap-154
> opensuse-tumbleweed
> ubuntu-1804
> ubuntu-2004
> ubuntu-2204
> 
> Contributors can then use 'make docker-XXXX' to run build tests
> locally on specific distros when they need to test something
> that isn't covered by default in out gating CI
> 
> 
> With regards,
> Daniel

[PATCH 08/12] tests: skip the nios2 replay_kernel test

[Alex Bennée]

It is buggy and keeps failing.

Suggested-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/avocado/replay_kernel.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py
index 00a26e4a0c..f13456e1ec 100644
--- a/tests/avocado/replay_kernel.py
+++ b/tests/avocado/replay_kernel.py
@@ -349,6 +349,7 @@ def test_or1k_sim(self):
         file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
         self.do_test_advcal_2018(file_path, 'vmlinux')
 
+    @skip("nios2 emulation is buggy under record/replay")
     def test_nios2_10m50(self):
         """
         :avocado: tags=arch:nios2
-- 
2.39.1

Re: [PATCH 08/12] tests: skip the nios2 replay_kernel test

[Philippe Mathieu-Daudé]

On 15/2/23 20:25, Alex Bennée wrote:
> It is buggy and keeps failing.
> 
> Suggested-by: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   tests/avocado/replay_kernel.py | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py
> index 00a26e4a0c..f13456e1ec 100644
> --- a/tests/avocado/replay_kernel.py
> +++ b/tests/avocado/replay_kernel.py
> @@ -349,6 +349,7 @@ def test_or1k_sim(self):
>           file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
>           self.do_test_advcal_2018(file_path, 'vmlinux')
>   
> +    @skip("nios2 emulation is buggy under record/replay")

It will be hard to notice the flakyness got improved. What about:

        @skipUnless(os.getenv('AVOCADO_RUN_FLAKY_TESTS'), 'Flaky test')

>       def test_nios2_10m50(self):
>           """
>           :avocado: tags=arch:nios2

Re: [PATCH 08/12] tests: skip the nios2 replay_kernel test

[Richard Henderson]

On 2/15/23 10:47, Philippe Mathieu-Daudé wrote:
>> +    @skip("nios2 emulation is buggy under record/replay")
> 
> It will be hard to notice the flakyness got improved. What about:
> 
>         @skipUnless(os.getenv('AVOCADO_RUN_FLAKY_TESTS'), 'Flaky test')
> 
>>       def test_nios2_10m50(self):
>>           """
>>           :avocado: tags=arch:nios2

It'll be hard for there to be any improvement without an active maintainer.

Last patch from any of the listed maintainers was:

commit ebedf0f9cd46b617df331eecc857c379d574ac62
Author: Marek Vasut <marex@denx.de>
Date:   Fri Mar 17 22:06:27 2017 +0100

     nios2: iic: Convert CPU prop to qom link

It's probably time to demote "Maintained" to "Orphan".


r~

Re: [PATCH 08/12] tests: skip the nios2 replay_kernel test

[Philippe Mathieu-Daudé]

On 15/2/23 21:59, Richard Henderson wrote:
> On 2/15/23 10:47, Philippe Mathieu-Daudé wrote:
>>> +    @skip("nios2 emulation is buggy under record/replay")
>>
>> It will be hard to notice the flakyness got improved. What about:
>>
>>         @skipUnless(os.getenv('AVOCADO_RUN_FLAKY_TESTS'), 'Flaky test')
>>
>>>       def test_nios2_10m50(self):
>>>           """
>>>           :avocado: tags=arch:nios2
> 
> It'll be hard for there to be any improvement without an active maintainer.

In that case no point in keeping the test...

Anyhow,
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

> Last patch from any of the listed maintainers was:
> 
> commit ebedf0f9cd46b617df331eecc857c379d574ac62
> Author: Marek Vasut <marex@denx.de>
> Date:   Fri Mar 17 22:06:27 2017 +0100
> 
>      nios2: iic: Convert CPU prop to qom link
> 
> It's probably time to demote "Maintained" to "Orphan".
> 
> 
> r~

Re: [PATCH 08/12] tests: skip the nios2 replay_kernel test

[Richard Henderson]

On 2/15/23 09:25, Alex Bennée wrote:
> It is buggy and keeps failing.
> 
> Suggested-by: Peter Maydell<peter.maydell@linaro.org>
> Signed-off-by: Alex Bennée<alex.bennee@linaro.org>
> ---
>   tests/avocado/replay_kernel.py | 1 +
>   1 file changed, 1 insertion(+)

Acked-by: Richard Henderson <richard.henderson@linaro.org>

r~

[PATCH 09/12] tests: add tuxrun baseline test to avocado

[Alex Bennée]

The TuxRun project (www.tuxrun.org) uses QEMU to run tests on a wide
variety of kernel configurations on wide range of our emulated
platforms. They publish a known good set of images at:

  https://storage.tuxboot.com/

to help with bisecting regressions in either the kernel, firmware or
QEMU itself. The tests are pretty lightweight as they contain just a
kernel with a minimal rootfs which boots a lot faster than most of the
distros. In time they might be persuaded to version there known good
baselines and we can then enable proper checksums.

For a couple of tests we currently skip:

  - mips64, a regression against previous stable release
  - sh4, very unstable with intermittent oops

Total run time: 340s (default) -> 890s (debug)

Overall coverage rate (tested targets + disabled tests):
  lines......: 16.1% (126894 of 789848 lines)
  functions..: 20.6% (15954 of 77489 functions)
  branches...: 9.3% (40727 of 439365 branches)

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Anders Roxell <anders.roxell@linaro.org>

---
v2
  - renamed to tuxrun_baselines, update commit message
  - add remaining targets
  - add more metadata tags for the differences
  - refactor tag code
  - skip mips64 and sh4 tests in CI
  - slightly increase delay for login
  - include in MAINTAINERS
---
 MAINTAINERS                       |   1 +
 tests/avocado/tuxrun_baselines.py | 423 ++++++++++++++++++++++++++++++
 2 files changed, 424 insertions(+)
 create mode 100644 tests/avocado/tuxrun_baselines.py

diff --git a/MAINTAINERS b/MAINTAINERS
index fd54c1f140..be100272b3 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3760,6 +3760,7 @@ F: scripts/ci/
 F: tests/docker/
 F: tests/vm/
 F: tests/lcitool/
+F: tests/avocado/tuxrun_baselines.py
 F: scripts/archive-source.sh
 F: docs/devel/testing.rst
 W: https://gitlab.com/qemu-project/qemu/pipelines
diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py
new file mode 100644
index 0000000000..30aaefc1d3
--- /dev/null
+++ b/tests/avocado/tuxrun_baselines.py
@@ -0,0 +1,423 @@
+# Functional test that boots known good tuxboot images the same way
+# that tuxrun (www.tuxrun.org) does. This tool is used by things like
+# the LKFT project to run regression tests on kernels.
+#
+# Copyright (c) 2023 Linaro Ltd.
+#
+# Author:
+#  Alex Bennée <alex.bennee@linaro.org>
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import os
+import time
+
+from avocado import skip, skipIf
+from avocado_qemu import QemuSystemTest
+from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
+from avocado_qemu import wait_for_console_pattern
+from avocado.utils import process
+from avocado.utils.path import find_command
+
+class TuxRunBaselineTest(QemuSystemTest):
+    """
+    :avocado: tags=accel:tcg
+    """
+
+    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0'
+    # Tests are ~10-40s, allow for --debug/--enable-gcov overhead
+    timeout = 100
+
+    def get_tag(self, tagname, default=None):
+        """
+        Get the metadata tag or return the default.
+        """
+        utag = self._get_unique_tag_val(tagname)
+        print(f"{tagname}/{default} -> {utag}")
+        if utag:
+            return utag
+
+        return default
+
+    def setUp(self):
+        super().setUp()
+
+        # We need zstd for all the tuxrun tests
+        # See https://github.com/avocado-framework/avocado/issues/5609
+        zstd = find_command('zstd', False)
+        if zstd is False:
+            self.cancel('Could not find "zstd", which is required to '
+                        'decompress rootfs')
+        self.zstd = zstd
+
+        # Process the TuxRun specific tags, most machines work with
+        # reasonable defaults but we sometimes need to tweak the
+        # config. To avoid open coding everything we store all these
+        # details in the metadata for each test.
+
+        # The tuxboot tag matches the root directory
+        self.tuxboot = self.get_tag('tuxboot')
+
+        # Most Linux's use ttyS0 for their serial port
+        self.console = self.get_tag('console', "ttyS0")
+
+        # Does the machine shutdown QEMU nicely on "halt"
+        self.shutdown = self.get_tag('shutdown')
+
+        # The name of the kernel Image file
+        self.image = self.get_tag('image', "Image")
+
+        # The block device drive type
+        self.drive = self.get_tag('drive', "virtio-blk-device")
+
+        self.root = self.get_tag('root', "vda")
+
+        # Occasionally we need extra devices to hook things up
+        self.extradev = self.get_tag('extradev')
+
+    def wait_for_console_pattern(self, success_message, vm=None):
+        wait_for_console_pattern(self, success_message,
+                                 failure_message='Kernel panic - not syncing',
+                                 vm=vm)
+
+    def fetch_tuxrun_assets(self, dt=None):
+        """
+        Fetch the TuxBoot assets. They are stored in a standard way so we
+        use the per-test tags to fetch details.
+        """
+        base_url = f"https://storage.tuxboot.com/{self.tuxboot}/"
+        kernel_image =  self.fetch_asset(base_url + self.image)
+        disk_image_zst = self.fetch_asset(base_url + "rootfs.ext4.zst")
+
+        cmd = f"{self.zstd} -d {disk_image_zst} -o {self.workdir}/rootfs.ext4"
+        process.run(cmd)
+
+        if dt:
+            dtb = self.fetch_asset(base_url + dt)
+        else:
+            dtb = None
+
+        return (kernel_image, self.workdir + "/rootfs.ext4", dtb)
+
+    def prepare_run(self, kernel, disk, dtb=None, console_index=0):
+        """
+        Setup to run and add the common parameters to the system
+        """
+        self.vm.set_console(console_index=console_index)
+
+        # all block devices are raw ext4's
+        blockdev = "driver=raw,file.driver=file," \
+            + f"file.filename={disk},node-name=hd0"
+
+        kcmd_line = self.KERNEL_COMMON_COMMAND_LINE
+        kcmd_line += f" root=/dev/{self.root}"
+        kcmd_line += f" console={self.console}"
+
+        self.vm.add_args('-kernel', kernel,
+                         '-append', kcmd_line,
+                         '-blockdev', blockdev)
+
+        # Sometimes we need extra devices attached
+        if self.extradev:
+            self.vm.add_args('-device', self.extradev)
+
+        # Some machines already define a drive device
+        if self.drive != "none":
+            self.vm.add_args('-device',
+                             f"{self.drive},drive=hd0")
+
+        # Some machines need an explicit DTB
+        if dtb:
+            self.vm.add_args('-dtb', dtb)
+
+    def run_tuxtest_tests(self, haltmsg):
+        """
+        Wait for the system to boot up, wait for the login prompt and
+        then do a few things on the console. Trigger a shutdown and
+        wait to exit cleanly.
+        """
+        self.wait_for_console_pattern("Welcome to TuxTest")
+        time.sleep(0.2)
+        exec_command(self, 'root')
+        time.sleep(0.2)
+        exec_command(self, 'cat /proc/interrupts')
+        time.sleep(0.1)
+        exec_command(self, 'cat /proc/self/maps')
+        time.sleep(0.1)
+        exec_command(self, 'uname -a')
+        time.sleep(0.1)
+        exec_command_and_wait_for_pattern(self, 'halt', haltmsg)
+
+        # Wait for VM to shut down gracefully if it can
+        if self.shutdown == "nowait":
+            self.vm.shutdown()
+        else:
+            self.vm.wait()
+
+    def common_tuxrun(self, dt=None, haltmsg="reboot: System halted",
+                      console_index=0):
+        """
+        Common path for LKFT tests. Unless we need to do something
+        special with the command line we can process most things using
+        the tag metadata.
+        """
+        (kernel, disk, dtb) = self.fetch_tuxrun_assets(dt)
+
+        self.prepare_run(kernel, disk, dtb, console_index)
+        self.vm.launch()
+        self.run_tuxtest_tests(haltmsg)
+
+    #
+    # The tests themselves. The configuration is derived from how
+    # tuxrun invokes qemu (with minor tweaks like using -blockdev
+    # consistently). The tuxrun equivalent is something like:
+    #
+    # tuxrun --device qemu-{ARCH} \
+    #        --kernel https://storage.tuxboot.com/{TUXBOOT}/{IMAGE}
+    #
+
+    def test_arm64(self):
+        """
+        :avocado: tags=arch:aarch64
+        :avocado: tags=cpu:cortex-a57
+        :avocado: tags=machine:virt
+        :avocado: tags=tuxboot:arm64
+        :avocado: tags=console:ttyAMA0
+        :avocado: tags=shutdown:nowait
+        """
+        self.common_tuxrun()
+
+    def test_arm64be(self):
+        """
+        :avocado: tags=arch:aarch64
+        :avocado: tags=cpu:cortex-a57
+        :avocado: tags=endian:big
+        :avocado: tags=machine:virt
+        :avocado: tags=tuxboot:arm64be
+        :avocado: tags=console:ttyAMA0
+        :avocado: tags=shutdown:nowait
+        """
+        self.common_tuxrun()
+
+    def test_armv5(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=cpu:arm926
+        :avocado: tags=machine:versatilepb
+        :avocado: tags=tuxboot:armv5
+        :avocado: tags=image:zImage
+        :avocado: tags=drive:virtio-blk-pci
+        :avocado: tags=console:ttyAMA0
+        :avocado: tags=shutdown:nowait
+        """
+        self.common_tuxrun(dt="versatile-pb.dtb")
+
+    def test_armv7(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=cpu:cortex-a15
+        :avocado: tags=machine:virt
+        :avocado: tags=tuxboot:armv7
+        :avocado: tags=image:zImage
+        :avocado: tags=console:ttyAMA0
+        :avocado: tags=shutdown:nowait
+        """
+        self.common_tuxrun()
+
+    def test_armv7be(self):
+        """
+        :avocado: tags=arch:arm
+        :avocado: tags=cpu:cortex-a15
+        :avocado: tags=endian:big
+        :avocado: tags=machine:virt
+        :avocado: tags=tuxboot:armv7be
+        :avocado: tags=image:zImage
+        :avocado: tags=console:ttyAMA0
+        :avocado: tags=shutdown:nowait
+        """
+        self.common_tuxrun()
+
+    def test_i386(self):
+        """
+        :avocado: tags=arch:i386
+        :avocado: tags=cpu:coreduo
+        :avocado: tags=machine:q35
+        :avocado: tags=tuxboot:i386
+        :avocado: tags=image:bzImage
+        :avocado: tags=drive:virtio-blk-pci
+        :avocado: tags=shutdown:nowait
+        """
+        self.common_tuxrun()
+
+    def test_mips32(self):
+        """
+        :avocado: tags=arch:mips
+        :avocado: tags=machine:malta
+        :avocado: tags=cpu:mips32r6-generic
+        :avocado: tags=endian:big
+        :avocado: tags=tuxboot:mips32
+        :avocado: tags=image:vmlinux
+        :avocado: tags=drive:driver=ide-hd,bus=ide.0,unit=0
+        :avocado: tags=root:sda
+        :avocado: tags=shutdown:nowait
+        """
+        self.common_tuxrun()
+
+    def test_mips32el(self):
+        """
+        :avocado: tags=arch:mipsel
+        :avocado: tags=machine:malta
+        :avocado: tags=cpu:mips32r6-generic
+        :avocado: tags=tuxboot:mips32el
+        :avocado: tags=image:vmlinux
+        :avocado: tags=drive:driver=ide-hd,bus=ide.0,unit=0
+        :avocado: tags=root:sda
+        :avocado: tags=shutdown:nowait
+        """
+        self.common_tuxrun()
+
+    @skip("QEMU currently broken") # regression against stable QEMU
+    def test_mips64(self):
+        """
+        :avocado: tags=arch:mips64
+        :avocado: tags=machine:malta
+        :avocado: tags=tuxboot:mips64
+        :avocado: tags=endian:big
+        :avocado: tags=image:vmlinux
+        :avocado: tags=drive:driver=ide-hd,bus=ide.0,unit=0
+        :avocado: tags=root:sda
+        :avocado: tags=shutdown:nowait
+        """
+        self.common_tuxrun()
+
+    def test_mips64el(self):
+        """
+        :avocado: tags=arch:mips64el
+        :avocado: tags=machine:malta
+        :avocado: tags=tuxboot:mips64el
+        :avocado: tags=image:vmlinux
+        :avocado: tags=drive:driver=ide-hd,bus=ide.0,unit=0
+        :avocado: tags=root:sda
+        :avocado: tags=shutdown:nowait
+        """
+        self.common_tuxrun()
+
+    def test_ppc32(self):
+        """
+        :avocado: tags=arch:ppc
+        :avocado: tags=machine:ppce500
+        :avocado: tags=cpu:e500mc
+        :avocado: tags=tuxboot:ppc32
+        :avocado: tags=image:uImage
+        :avocado: tags=drive:virtio-blk-pci
+        :avocado: tags=shutdown:nowait
+        """
+        self.common_tuxrun()
+
+    def test_ppc64(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:pseries
+        :avocado: tags=cpu:POWER8
+        :avocado: tags=endian:big
+        :avocado: tags=console:hvc0
+        :avocado: tags=tuxboot:ppc64
+        :avocado: tags=image:vmlinux
+        :avocado: tags=extradev:driver=spapr-vscsi
+        :avocado: tags=drive:scsi-hd
+        :avocado: tags=root:sda
+        """
+        self.common_tuxrun()
+
+    def test_ppc64le(self):
+        """
+        :avocado: tags=arch:ppc64
+        :avocado: tags=machine:pseries
+        :avocado: tags=cpu:POWER8
+        :avocado: tags=console:hvc0
+        :avocado: tags=tuxboot:ppc64le
+        :avocado: tags=image:vmlinux
+        :avocado: tags=extradev:driver=spapr-vscsi
+        :avocado: tags=drive:scsi-hd
+        :avocado: tags=root:sda
+        """
+        self.common_tuxrun()
+
+    def test_riscv32(self):
+        """
+        :avocado: tags=arch:riscv32
+        :avocado: tags=machine:virt
+        :avocado: tags=tuxboot:riscv32
+        """
+        self.common_tuxrun()
+
+    def test_riscv64(self):
+        """
+        :avocado: tags=arch:riscv64
+        :avocado: tags=machine:virt
+        :avocado: tags=tuxboot:riscv64
+        """
+        self.common_tuxrun()
+
+    def test_s390(self):
+        """
+        :avocado: tags=arch:s390x
+        :avocado: tags=endian:big
+        :avocado: tags=tuxboot:s390
+        :avocado: tags=image:bzImage
+        :avocado: tags=drive:virtio-blk-ccw
+        :avocado: tags=shutdown:nowait
+        """
+        self.common_tuxrun(haltmsg="Requesting system halt")
+
+    # Note: some segfaults caused by unaligned userspace access
+    @skipIf(os.getenv('GITLAB_CI'), 'Skipping unstable test on GitLab')
+    def test_sh4(self):
+        """
+        :avocado: tags=arch:sh4
+        :avocado: tags=machine:r2d
+        :avocado: tags=cpu:sh7785
+        :avocado: tags=tuxboot:sh4
+        :avocado: tags=image:zImage
+        :avocado: tags=root:sda
+        :avocado: tags=drive:driver=ide-hd,bus=ide.0,unit=0
+        :avocado: tags=console:ttySC1
+        """
+        # The test is currently too unstable to do much in userspace
+        # so we skip common_tuxrun and do a minimal boot and shutdown.
+        (kernel, disk, dtb) = self.fetch_tuxrun_assets()
+
+        # the console comes on the second serial port
+        self.prepare_run(kernel, disk, console_index=1)
+        self.vm.launch()
+
+        self.wait_for_console_pattern("Welcome to TuxTest")
+        time.sleep(0.1)
+        exec_command(self, 'root')
+        time.sleep(0.1)
+        exec_command_and_wait_for_pattern(self, 'halt',
+                                          "reboot: System halted")
+
+    def test_sparc64(self):
+        """
+        :avocado: tags=arch:sparc64
+        :avocado: tags=tuxboot:sparc64
+        :avocado: tags=image:vmlinux
+        :avocado: tags=root:sda
+        :avocado: tags=drive:driver=ide-hd,bus=ide.0,unit=0
+        :avocado: tags=shutdown:nowait
+        """
+        self.common_tuxrun()
+
+    def test_x86_64(self):
+        """
+        :avocado: tags=arch:x86_64
+        :avocado: tags=machine:q35
+        :avocado: tags=cpu:Nehalem
+        :avocado: tags=tuxboot:x86_64
+        :avocado: tags=image:bzImage
+        :avocado: tags=root:sda
+        :avocado: tags=drive:driver=ide-hd,bus=ide.0,unit=0
+        :avocado: tags=shutdown:nowait
+        """
+        self.common_tuxrun()
-- 
2.39.1

Re: [PATCH 09/12] tests: add tuxrun baseline test to avocado

[Philippe Mathieu-Daudé]

On 15/2/23 20:25, Alex Bennée wrote:
> The TuxRun project (www.tuxrun.org) uses QEMU to run tests on a wide
> variety of kernel configurations on wide range of our emulated
> platforms. They publish a known good set of images at:
> 
>    https://storage.tuxboot.com/
> 
> to help with bisecting regressions in either the kernel, firmware or
> QEMU itself. The tests are pretty lightweight as they contain just a
> kernel with a minimal rootfs which boots a lot faster than most of the
> distros. In time they might be persuaded to version there known good
> baselines and we can then enable proper checksums.
> 
> For a couple of tests we currently skip:
> 
>    - mips64, a regression against previous stable release
>    - sh4, very unstable with intermittent oops
> 
> Total run time: 340s (default) -> 890s (debug)
> 
> Overall coverage rate (tested targets + disabled tests):
>    lines......: 16.1% (126894 of 789848 lines)
>    functions..: 20.6% (15954 of 77489 functions)
>    branches...: 9.3% (40727 of 439365 branches)
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Anders Roxell <anders.roxell@linaro.org>
> 
> ---
> v2
>    - renamed to tuxrun_baselines, update commit message
>    - add remaining targets
>    - add more metadata tags for the differences
>    - refactor tag code
>    - skip mips64 and sh4 tests in CI
>    - slightly increase delay for login
>    - include in MAINTAINERS
> ---
>   MAINTAINERS                       |   1 +
>   tests/avocado/tuxrun_baselines.py | 423 ++++++++++++++++++++++++++++++
>   2 files changed, 424 insertions(+)
>   create mode 100644 tests/avocado/tuxrun_baselines.py
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index fd54c1f140..be100272b3 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -3760,6 +3760,7 @@ F: scripts/ci/
>   F: tests/docker/
>   F: tests/vm/
>   F: tests/lcitool/
> +F: tests/avocado/tuxrun_baselines.py
>   F: scripts/archive-source.sh
>   F: docs/devel/testing.rst
>   W: https://gitlab.com/qemu-project/qemu/pipelines
> diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py
> new file mode 100644
> index 0000000000..30aaefc1d3
> --- /dev/null
> +++ b/tests/avocado/tuxrun_baselines.py
> @@ -0,0 +1,423 @@
> +# Functional test that boots known good tuxboot images the same way
> +# that tuxrun (www.tuxrun.org) does. This tool is used by things like
> +# the LKFT project to run regression tests on kernels.
> +#
> +# Copyright (c) 2023 Linaro Ltd.
> +#
> +# Author:
> +#  Alex Bennée <alex.bennee@linaro.org>
> +#
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +
> +import os
> +import time
> +
> +from avocado import skip, skipIf
> +from avocado_qemu import QemuSystemTest
> +from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
> +from avocado_qemu import wait_for_console_pattern
> +from avocado.utils import process
> +from avocado.utils.path import find_command
> +
> +class TuxRunBaselineTest(QemuSystemTest):

Better inherit from LinuxKernelTest, adding a zstd method there.

> +    """
> +    :avocado: tags=accel:tcg
> +    """
> +
> +    KERNEL_COMMON_COMMAND_LINE = 'printk.time=0'
> +    # Tests are ~10-40s, allow for --debug/--enable-gcov overhead
> +    timeout = 100
> +
> +    def get_tag(self, tagname, default=None):
> +        """
> +        Get the metadata tag or return the default.
> +        """
> +        utag = self._get_unique_tag_val(tagname)
> +        print(f"{tagname}/{default} -> {utag}")
> +        if utag:
> +            return utag
> +
> +        return default
> +
> +    def setUp(self):
> +        super().setUp()
> +
> +        # We need zstd for all the tuxrun tests
> +        # See https://github.com/avocado-framework/avocado/issues/5609
> +        zstd = find_command('zstd', False)
> +        if zstd is False:
> +            self.cancel('Could not find "zstd", which is required to '
> +                        'decompress rootfs')
> +        self.zstd = zstd
> +
> +        # Process the TuxRun specific tags, most machines work with
> +        # reasonable defaults but we sometimes need to tweak the
> +        # config. To avoid open coding everything we store all these
> +        # details in the metadata for each test.
> +
> +        # The tuxboot tag matches the root directory
> +        self.tuxboot = self.get_tag('tuxboot')
> +
> +        # Most Linux's use ttyS0 for their serial port
> +        self.console = self.get_tag('console', "ttyS0")
> +
> +        # Does the machine shutdown QEMU nicely on "halt"
> +        self.shutdown = self.get_tag('shutdown')
> +
> +        # The name of the kernel Image file
> +        self.image = self.get_tag('image', "Image")
> +
> +        # The block device drive type
> +        self.drive = self.get_tag('drive', "virtio-blk-device")
> +
> +        self.root = self.get_tag('root', "vda")
> +
> +        # Occasionally we need extra devices to hook things up
> +        self.extradev = self.get_tag('extradev')
> +
> +    def wait_for_console_pattern(self, success_message, vm=None):
> +        wait_for_console_pattern(self, success_message,
> +                                 failure_message='Kernel panic - not syncing',
> +                                 vm=vm)

(we'd then inherit this method from LinuxKernelTest)

Whichever outcome you rather:
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

[PATCH 10/12] tests/docker: Use binaries for debian-tricore-cross

[Alex Bennée]

From: Bastian Koppelmann <kbastian@mail.uni-paderborn.de>

since binutils is pretty old, it fails our CI repeatedly during the
compilation of tricore-binutils. We created a precompiled version using
the debian docker image and download it instead of building it ourself.

We also updated the package to include a newer version of binutils, gcc,
and newlib. The default TriCore ISA version used by tricore-as changed
from the old version, so we have to specify it now. If we don't
'test_fadd' fails with 'unknown opcode'.

The new assembler also picks a new encoding in ld.h which fails the
'test_ld_h' test. We fix that by using the newest TriCore CPU for QEMU.

The old assembler accepted an extra ')' in 'test_imask'. The new one
does not, so lets remove it.

Signed-off-by: Bastian Koppelmann <kbastian@mail.uni-paderborn.de>
Message-Id: <20230209145812.46730-1-kbastian@mail.uni-paderborn.de>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/tcg/tricore/macros.h                           |  2 +-
 tests/docker/dockerfiles/debian-tricore-cross.docker | 10 +++-------
 tests/tcg/tricore/Makefile.softmmu-target            |  6 +++---
 3 files changed, 7 insertions(+), 11 deletions(-)

diff --git a/tests/tcg/tricore/macros.h b/tests/tcg/tricore/macros.h
index ec4f5bff52..3df2e0de82 100644
--- a/tests/tcg/tricore/macros.h
+++ b/tests/tcg/tricore/macros.h
@@ -174,7 +174,7 @@ test_ ## num:                                                    \
     TEST_CASE_E(num, res_lo, res_hi,                           \
     LI(DREG_RS1, rs1);                                         \
     rstv;                                                      \
-    insn EREG_CALC_RESULT, imm1, DREG_RS1, imm2);              \
+    insn EREG_CALC_RESULT, imm1, DREG_RS1, imm2;               \
     )
 
 
diff --git a/tests/docker/dockerfiles/debian-tricore-cross.docker b/tests/docker/dockerfiles/debian-tricore-cross.docker
index 5ae58efa09..82e4576485 100644
--- a/tests/docker/dockerfiles/debian-tricore-cross.docker
+++ b/tests/docker/dockerfiles/debian-tricore-cross.docker
@@ -20,6 +20,7 @@ RUN apt update && \
        bzip2 \
        ca-certificates \
        ccache \
+       curl \
        flex \
        g++ \
        gcc \
@@ -34,13 +35,8 @@ RUN apt update && \
        python3-setuptools \
        python3-wheel
 
-RUN git clone --single-branch \
-        https://github.com/bkoppelmann/tricore-binutils.git \
-        /usr/src/binutils && \
-    cd /usr/src/binutils && chmod +x missing && \
-    CFLAGS=-w ./configure --prefix=/usr/local --disable-nls --target=tricore && \
-    make && make install && \
-    rm -rf /usr/src/binutils
+RUN curl -#SL https://github.com/bkoppelmann/package_940/releases/download/tricore-toolchain-9.40/tricore-toolchain-9.4.0.tar.gz \
+    | tar -xzC /usr/local/
 
 # This image can only build a very minimal QEMU as well as the tests
 ENV DEF_TARGET_LIST tricore-softmmu
diff --git a/tests/tcg/tricore/Makefile.softmmu-target b/tests/tcg/tricore/Makefile.softmmu-target
index d2446af8b4..b3cd56fffc 100644
--- a/tests/tcg/tricore/Makefile.softmmu-target
+++ b/tests/tcg/tricore/Makefile.softmmu-target
@@ -1,7 +1,7 @@
 TESTS_PATH = $(SRC_PATH)/tests/tcg/tricore
 
-LDFLAGS = -T$(TESTS_PATH)/link.ld
-ASFLAGS =
+LDFLAGS = -T$(TESTS_PATH)/link.ld --mcpu=tc162
+ASFLAGS = -mtc162
 
 TESTS += test_abs.tst
 TESTS += test_bmerge.tst
@@ -19,7 +19,7 @@ TESTS += test_madd.tst
 TESTS += test_msub.tst
 TESTS += test_muls.tst
 
-QEMU_OPTS += -M tricore_testboard -nographic -kernel
+QEMU_OPTS += -M tricore_testboard -cpu tc27x -nographic -kernel
 
 %.pS: $(TESTS_PATH)/%.S
 	$(HOST_CC) -E -o $@ $<
-- 
2.39.1

Re: [PATCH 10/12] tests/docker: Use binaries for debian-tricore-cross

[Philippe Mathieu-Daudé]

On 15/2/23 20:25, Alex Bennée wrote:
> From: Bastian Koppelmann <kbastian@mail.uni-paderborn.de>
> 
> since binutils is pretty old, it fails our CI repeatedly during the
> compilation of tricore-binutils. We created a precompiled version using
> the debian docker image and download it instead of building it ourself.
> 
> We also updated the package to include a newer version of binutils, gcc,
> and newlib. The default TriCore ISA version used by tricore-as changed
> from the old version, so we have to specify it now. If we don't
> 'test_fadd' fails with 'unknown opcode'.
> 
> The new assembler also picks a new encoding in ld.h which fails the
> 'test_ld_h' test. We fix that by using the newest TriCore CPU for QEMU.
> 
> The old assembler accepted an extra ')' in 'test_imask'. The new one
> does not, so lets remove it.
> 
> Signed-off-by: Bastian Koppelmann <kbastian@mail.uni-paderborn.de>
> Message-Id: <20230209145812.46730-1-kbastian@mail.uni-paderborn.de>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   tests/tcg/tricore/macros.h                           |  2 +-
>   tests/docker/dockerfiles/debian-tricore-cross.docker | 10 +++-------
>   tests/tcg/tricore/Makefile.softmmu-target            |  6 +++---
>   3 files changed, 7 insertions(+), 11 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

[PATCH 11/12] cirrus.yml: Improve the windows_msys2_task

[Alex Bennée]

From: Thomas Huth <thuth@redhat.com>

There's no need to run a full-blown bash just to create a directory.
And we can skip the "cd build" each time by doing it once at the
beginning.

Additionally, let's exclude some targets (that we already compile-test
with MinGW in the gitlab jobs) from the build, since the build time of
this task is very long already (between 80 and 90 minutes).

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230208103046.618154-1-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .cirrus.yml | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 4895987da4..5fb00da73d 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -100,9 +100,11 @@ windows_msys2_task:
       tar xf C:\tools\archive\msys64.tar
       Write-Output "Extract msys2 time taken: $((Get-Date).Subtract($start_time))"
   script:
-    - C:\tools\msys64\usr\bin\bash.exe -lc "mkdir build"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && ../configure --python=python3"
-    - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make -j8"
+    - mkdir build
+    - cd build
+    - C:\tools\msys64\usr\bin\bash.exe -lc "../configure --python=python3
+        --target-list-exclude=i386-softmmu,ppc64-softmmu,aarch64-softmmu,mips64-softmmu,mipsel-softmmu,sh4-softmmu"
+    - C:\tools\msys64\usr\bin\bash.exe -lc "make -j8"
     - exit $LastExitCode
   test_script:
     - C:\tools\msys64\usr\bin\bash.exe -lc "cd build && make V=1 check"
-- 
2.39.1

Re: [PATCH 11/12] cirrus.yml: Improve the windows_msys2_task

[Philippe Mathieu-Daudé]

On 15/2/23 20:25, Alex Bennée wrote:
> From: Thomas Huth <thuth@redhat.com>
> 
> There's no need to run a full-blown bash just to create a directory.
> And we can skip the "cd build" each time by doing it once at the
> beginning.
> 
> Additionally, let's exclude some targets (that we already compile-test
> with MinGW in the gitlab jobs) from the build, since the build time of
> this task is very long already (between 80 and 90 minutes).
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> Message-Id: <20230208103046.618154-1-thuth@redhat.com>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   .cirrus.yml | 8 +++++---
>   1 file changed, 5 insertions(+), 3 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

[PATCH 12/12] tests: ensure we export job results for some cross builds

[Alex Bennée]

We do run tests on some cross builds. Provide a template to ensure we
export the testlog to the build artefacts and report the test results
via the junit.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reported-by: Peter Maydell <peter.maydell@linaro.org>

---
v2
  - properly format extends
---
 .gitlab-ci.d/crossbuild-template.yml | 11 +++++++++++
 .gitlab-ci.d/crossbuilds.yml         | 12 +++++++++---
 2 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
index 6d709628f1..24a41a7b21 100644
--- a/.gitlab-ci.d/crossbuild-template.yml
+++ b/.gitlab-ci.d/crossbuild-template.yml
@@ -48,3 +48,14 @@
           nios2-linux-user or1k-linux-user ppc-linux-user sparc-linux-user
           xtensa-linux-user $CROSS_SKIP_TARGETS"
     - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
+
+# We can still run some tests on some of our cross build jobs. They can add this
+# template to their extends to save the build logs and test results
+.cross_test_artifacts:
+  artifacts:
+    name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
+    expire_in: 7 days
+    paths:
+      - build/meson-logs/testlog.txt
+    reports:
+      junit: build/meson-logs/testlog.junit.xml
diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index 74d6259b90..bbc013725c 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -44,7 +44,9 @@ cross-arm64-user:
     IMAGE: debian-arm64-cross
 
 cross-i386-system:
-  extends: .cross_system_build_job
+  extends:
+    - .cross_system_build_job
+    - .cross_test_artifacts
   needs:
     job: i386-fedora-cross-container
   variables:
@@ -52,7 +54,9 @@ cross-i386-system:
     MAKE_CHECK_ARGS: check-qtest
 
 cross-i386-user:
-  extends: .cross_user_build_job
+  extends:
+    - .cross_user_build_job
+    - .cross_test_artifacts
   needs:
     job: i386-fedora-cross-container
   variables:
@@ -60,7 +64,9 @@ cross-i386-user:
     MAKE_CHECK_ARGS: check
 
 cross-i386-tci:
-  extends: .cross_accel_build_job
+  extends:
+    - .cross_accel_build_job
+    - .cross_test_artifacts
   timeout: 60m
   needs:
     job: i386-fedora-cross-container
-- 
2.39.1

Re: [PATCH 12/12] tests: ensure we export job results for some cross builds

[Thomas Huth]

On 15/02/2023 20.25, Alex Bennée wrote:
> We do run tests on some cross builds. Provide a template to ensure we
> export the testlog to the build artefacts and report the test results
> via the junit.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Reported-by: Peter Maydell <peter.maydell@linaro.org>
> 
> ---
> v2
>    - properly format extends
> ---
>   .gitlab-ci.d/crossbuild-template.yml | 11 +++++++++++
>   .gitlab-ci.d/crossbuilds.yml         | 12 +++++++++---
>   2 files changed, 20 insertions(+), 3 deletions(-)

Reviewed-by: Thomas Huth <thuth@redhat.com>