[PATCH 00/13] Bump Avocado to 103.0 LTS and update tests for compatibility and new features

[PATCH 00/13] Bump Avocado to 103.0 LTS and update tests for compatibility and new features

[Cleber Rosa]

This is a *long* overdue update of the Avocado version used in QEMU.
It comes a time where the role of the runner and the libraries are
being discussed and questioned.

These exact commits have been staging on my side for over 30 days now,
and I was exceeding what I should in terms of testing before posting.
I apologize for the miscalculation.

Nevertheless, as pointed out, on the ML, these changes are needed NOW.

Some examples of runs in the CI can be seen below:

* Serial with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7074346143#L220):
   RESULTS    : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
   JOB TIME   : 432.63 s

* Parallel with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478#L222)
   RESULTS    : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
   JOB TIME   : 148.99 s

Cleber Rosa (13):
  tests/avocado: mips: fallback to HTTP given certificate expiration
  tests/avocado: mips: add hint for fetchasset plugin
  tests/avocado/intel_iommu.py: increase timeout
  tests/avocado: add cdrom permission related tests
  tests/avocado: machine aarch64: standardize location and RO access
  tests/avocado: use more distinct names for assets
  tests/avocado/kvm_xen_guest.py: cope with asset RW requirements
  testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset
  tests/avocado/boot_xen.py: fetch kernel during test setUp()
  tests/avocado/tuxrun_baselines.py: use Avocado's zstd support
  tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
  Bump avocado to 103.0
  Avocado tests: allow for parallel execution of tests

 docs/devel/testing.rst                   | 12 +++++++
 pythondeps.toml                          |  2 +-
 tests/Makefile.include                   |  6 +++-
 tests/avocado/boot_linux_console.py      | 24 ++++++++------
 tests/avocado/boot_xen.py                | 13 ++++----
 tests/avocado/cdrom.py                   | 41 ++++++++++++++++++++++++
 tests/avocado/intel_iommu.py             |  2 ++
 tests/avocado/kvm_xen_guest.py           | 30 +++++++++++------
 tests/avocado/machine_aarch64_sbsaref.py | 11 +++++--
 tests/avocado/machine_aarch64_virt.py    | 14 ++++----
 tests/avocado/netdev-ethtool.py          |  3 +-
 tests/avocado/tuxrun_baselines.py        | 16 ++++-----
 12 files changed, 125 insertions(+), 49 deletions(-)
 create mode 100644 tests/avocado/cdrom.py

-- 
2.45.2

[PATCH 01/13] tests/avocado: mips: fallback to HTTP given certificate expiration

[Cleber Rosa]

The SSL certificate installed at mipsdistros.mips.com has expired:

 0 s:CN = mipsdistros.mips.com
 i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
 a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
 v:NotBefore: Dec 23 00:00:00 2019 GMT; NotAfter: Jan 23 12:00:00 2021 GMT

Because this project has no control over that certificate and host,
this falls back to plain HTTP instead.  The integrity of the
downloaded files can be guaranteed by the existing hashes for those
files (which are not modified here).

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/boot_linux_console.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
index c35fc5e9ba..450d67be6a 100644
--- a/tests/avocado/boot_linux_console.py
+++ b/tests/avocado/boot_linux_console.py
@@ -299,7 +299,7 @@ def test_mips_malta32el_nanomips_4k(self):
         :avocado: tags=endian:little
         :avocado: tags=cpu:I7200
         """
-        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+        kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/'
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page4k.xz')
         kernel_hash = '477456aafd2a0f1ddc9482727f20fe9575565dd6'
@@ -312,7 +312,7 @@ def test_mips_malta32el_nanomips_16k_up(self):
         :avocado: tags=endian:little
         :avocado: tags=cpu:I7200
         """
-        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+        kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/'
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page16k_up.xz')
         kernel_hash = 'e882868f944c71c816e832e2303b7874d044a7bc'
@@ -325,7 +325,7 @@ def test_mips_malta32el_nanomips_64k_dbg(self):
         :avocado: tags=endian:little
         :avocado: tags=cpu:I7200
         """
-        kernel_url = ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
+        kernel_url = ('http://mipsdistros.mips.com/LinuxDistro/nanomips/'
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page64k_dbg.xz')
         kernel_hash = '18d1c68f2e23429e266ca39ba5349ccd0aeb7180'
-- 
2.45.2

[PATCH 02/13] tests/avocado: mips: add hint for fetchasset plugin

[Cleber Rosa]

Avocado's fetchasset plugin runs before the actual Avocado job (and
any test).  It analyses the test's code looking for occurrences of
"self.fetch_asset()" in the either the actual test or setUp() method.
It's not able to fully analyze all code, though.

The way these tests are written, make the fetchasset plugin blind to
the assets.  This adds some more code duplication, true, but it will
aid the fetchasset plugin to download or verify the existence of these
assets in advance.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/boot_linux_console.py | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
index 450d67be6a..b8b0a4df10 100644
--- a/tests/avocado/boot_linux_console.py
+++ b/tests/avocado/boot_linux_console.py
@@ -274,8 +274,7 @@ def test_mips64el_malta_5KEc_cpio(self):
         # Wait for VM to shut down gracefully
         self.vm.wait()
 
-    def do_test_mips_malta32el_nanomips(self, kernel_url, kernel_hash):
-        kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+    def do_test_mips_malta32el_nanomips(self, kernel_path_xz):
         kernel_path = self.workdir + "kernel"
         with lzma.open(kernel_path_xz, 'rb') as f_in:
             with open(kernel_path, 'wb') as f_out:
@@ -303,7 +302,8 @@ def test_mips_malta32el_nanomips_4k(self):
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page4k.xz')
         kernel_hash = '477456aafd2a0f1ddc9482727f20fe9575565dd6'
-        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
+        kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+        self.do_test_mips_malta32el_nanomips(kernel_path_xz)
 
     def test_mips_malta32el_nanomips_16k_up(self):
         """
@@ -316,7 +316,8 @@ def test_mips_malta32el_nanomips_16k_up(self):
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page16k_up.xz')
         kernel_hash = 'e882868f944c71c816e832e2303b7874d044a7bc'
-        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
+        kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+        self.do_test_mips_malta32el_nanomips(kernel_path_xz)
 
     def test_mips_malta32el_nanomips_64k_dbg(self):
         """
@@ -329,7 +330,8 @@ def test_mips_malta32el_nanomips_64k_dbg(self):
                       'kernels/v4.15.18-432-gb2eb9a8b07a1-20180627102142/'
                       'generic_nano32r6el_page64k_dbg.xz')
         kernel_hash = '18d1c68f2e23429e266ca39ba5349ccd0aeb7180'
-        self.do_test_mips_malta32el_nanomips(kernel_url, kernel_hash)
+        kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+        self.do_test_mips_malta32el_nanomips(kernel_path_xz)
 
     def test_aarch64_xlnx_versal_virt(self):
         """
-- 
2.45.2

[PATCH 03/13] tests/avocado/intel_iommu.py: increase timeout

[Cleber Rosa]

Based on many runs, the average run time for these 4 tests is around
250 seconds, with 320 seconds being the ceiling.  In any way, the
default 120 seconds timeout is inappropriate in my experience.

Let's increase the timeout so these tests get a chance to completion.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/intel_iommu.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/avocado/intel_iommu.py b/tests/avocado/intel_iommu.py
index 008f214397..9e7965c5df 100644
--- a/tests/avocado/intel_iommu.py
+++ b/tests/avocado/intel_iommu.py
@@ -25,6 +25,8 @@ class IntelIOMMU(LinuxTest):
     :avocado: tags=flaky
     """
 
+    timeout = 360
+
     IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'
     kernel_path = None
     initrd_path = None
-- 
2.45.2

[PATCH 04/13] tests/avocado: add cdrom permission related tests

[Cleber Rosa]

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/cdrom.py | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
 create mode 100644 tests/avocado/cdrom.py

diff --git a/tests/avocado/cdrom.py b/tests/avocado/cdrom.py
new file mode 100644
index 0000000000..c9aa5d69cb
--- /dev/null
+++ b/tests/avocado/cdrom.py
@@ -0,0 +1,41 @@
+# Simple functional tests for cdrom devices
+#
+# Copyright (c) 2023 Red Hat, Inc.
+#
+# Author:
+#  Cleber Rosa <crosa@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+
+from avocado.utils.iso9660 import iso9660
+from avocado_qemu import QemuSystemTest
+
+
+class Cdrom(QemuSystemTest):
+    """
+    :avocado: tags=block,cdrom,quick
+    :avocado: tags=machine:none
+    """
+    def setUp(self):
+        super().setUp()
+        self.iso_path = os.path.join(self.workdir, "cdrom.iso")
+        iso = iso9660(self.iso_path)
+        iso.create()
+        iso.close()
+
+    def test_plain_iso_rw(self):
+        self.vm.add_args('-drive', f'file={self.iso_path}')
+        self.vm.launch()
+        query_block_result = self.vm.qmp('query-block')['return']
+        self.assertEqual(len(query_block_result), 1)
+        self.assertFalse(query_block_result[0]["inserted"]["ro"])
+
+    def test_media_cdrom_ro(self):
+        self.vm.add_args('-drive', f'file={self.iso_path},media=cdrom')
+        self.vm.launch()
+        query_block_result = self.vm.qmp('query-block')['return']
+        self.assertEqual(len(query_block_result), 1)
+        self.assertTrue(query_block_result[0]["inserted"]["ro"])
-- 
2.45.2

[PATCH 05/13] tests/avocado: machine aarch64: standardize location and RO access

[Cleber Rosa]

The tests under machine_aarch64_virt.py and machine_aarch64_sbsaref.py
should not be writing to the ISO files.  By adding "media=cdrom" the
"ro" is autmatically set.

While at it, let's use a single code style and hash for the ISO url.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/machine_aarch64_sbsaref.py |  6 +++++-
 tests/avocado/machine_aarch64_virt.py    | 14 +++++++-------
 2 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
index e920bbf08c..1275f24532 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -129,7 +129,11 @@ def boot_alpine_linux(self, cpu):
             "-cpu",
             cpu,
             "-drive",
-            f"file={iso_path},format=raw",
+            f"file={iso_path},media=cdrom,format=raw",
+            "-device",
+            "virtio-rng-pci,rng=rng0",
+            "-object",
+            "rng-random,id=rng0,filename=/dev/urandom",
         )
 
         self.vm.launch()
diff --git a/tests/avocado/machine_aarch64_virt.py b/tests/avocado/machine_aarch64_virt.py
index a90dc6ff4b..6831d2c0ed 100644
--- a/tests/avocado/machine_aarch64_virt.py
+++ b/tests/avocado/machine_aarch64_virt.py
@@ -37,13 +37,13 @@ def test_alpine_virt_tcg_gic_max(self):
         :avocado: tags=machine:virt
         :avocado: tags=accel:tcg
         """
-        iso_url = ('https://dl-cdn.alpinelinux.org/'
-                   'alpine/v3.17/releases/aarch64/'
-                   'alpine-standard-3.17.2-aarch64.iso')
+        iso_url = (
+            "https://dl-cdn.alpinelinux.org/"
+            "alpine/v3.17/releases/aarch64/alpine-standard-3.17.2-aarch64.iso"
+        )
 
-        # Alpine use sha256 so I recalculated this myself
-        iso_sha1 = '76284fcd7b41fe899b0c2375ceb8470803eea839'
-        iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha1)
+        iso_hash = "5a36304ecf039292082d92b48152a9ec21009d3a62f459de623e19c4bd9dc027"
+        iso_path = self.fetch_asset(iso_url, algorithm="sha256", asset_hash=iso_hash)
 
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
@@ -60,7 +60,7 @@ def test_alpine_virt_tcg_gic_max(self):
         self.vm.add_args("-smp", "2", "-m", "1024")
         self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios',
                                                'edk2-aarch64-code.fd'))
-        self.vm.add_args("-drive", f"file={iso_path},format=raw")
+        self.vm.add_args("-drive", f"file={iso_path},media=cdrom,format=raw")
         self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
         self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
 
-- 
2.45.2

[PATCH 06/13] tests/avocado: use more distinct names for assets

[Cleber Rosa]

Avocado's asset system will deposit files in a cache organized either
by their original location (the URI) or by their names.  Because the
cache (and the "by_name" sub directory) is common across tests, it's a
good idea to make these names as distinct as possible.

This avoid name clashes, which makes future Avocado runs to attempt to
redownload the assets with the same name, but from the different
locations they actually are from.  This causes cache misses, extra
downloads, and possibly canceled tests.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/kvm_xen_guest.py  | 3 ++-
 tests/avocado/netdev-ethtool.py | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/avocado/kvm_xen_guest.py b/tests/avocado/kvm_xen_guest.py
index f8cb458d5d..318fadebc3 100644
--- a/tests/avocado/kvm_xen_guest.py
+++ b/tests/avocado/kvm_xen_guest.py
@@ -40,7 +40,8 @@ def get_asset(self, name, sha1):
         url = base_url + name
         # use explicit name rather than failing to neatly parse the
         # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
+                                locations=(url), asset_hash=sha1)
 
     def common_vm_setup(self):
         # We also catch lack of KVM_XEN support if we fail to launch
diff --git a/tests/avocado/netdev-ethtool.py b/tests/avocado/netdev-ethtool.py
index 5f33288f81..462cf8de7d 100644
--- a/tests/avocado/netdev-ethtool.py
+++ b/tests/avocado/netdev-ethtool.py
@@ -27,7 +27,8 @@ def get_asset(self, name, sha1):
         url = base_url + name
         # use explicit name rather than failing to neatly parse the
         # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(name=f"qemu-netdev-ethtool-{name}",
+                                locations=(url), asset_hash=sha1)
 
     def common_test_code(self, netdev, extra_args=None):
 
-- 
2.45.2

[PATCH 07/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements

[Cleber Rosa]

Some of these tests actually require the root filesystem image,
obtained through Avocado's asset feature and kept in a common cache
location, to be writable.

This makes a distinction between the tests that actually have this
requirement and those who don't.  The goal is to be as safe as
possible, avoiding causing cache misses (because the assets get
modified and thus need to be dowloaded again) while avoid copying the
root filesystem backing file whenever possible.

This also allow these tests to be run in parallel with newer Avocado
versions.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/kvm_xen_guest.py | 27 ++++++++++++++++++---------
 1 file changed, 18 insertions(+), 9 deletions(-)

diff --git a/tests/avocado/kvm_xen_guest.py b/tests/avocado/kvm_xen_guest.py
index 318fadebc3..d73fa888ef 100644
--- a/tests/avocado/kvm_xen_guest.py
+++ b/tests/avocado/kvm_xen_guest.py
@@ -10,6 +10,7 @@
 # SPDX-License-Identifier: GPL-2.0-or-later
 
 import os
+import shutil
 
 from qemu.machine import machine
 
@@ -43,7 +44,7 @@ def get_asset(self, name, sha1):
         return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
                                 locations=(url), asset_hash=sha1)
 
-    def common_vm_setup(self):
+    def common_vm_setup(self, readwrite=False):
         # We also catch lack of KVM_XEN support if we fail to launch
         self.require_accelerator("kvm")
 
@@ -56,11 +57,19 @@ def common_vm_setup(self):
                                           "367962983d0d32109998a70b45dcee4672d0b045")
         self.rootfs = self.get_asset("rootfs.ext4",
                                      "f1478401ea4b3fa2ea196396be44315bab2bb5e4")
+        if readwrite:
+            dest = os.path.join(self.workdir, os.path.basename(self.rootfs))
+            shutil.copy(self.rootfs, dest)
+            self.rootfs = dest
 
-    def run_and_check(self):
+    def run_and_check(self, readwrite=False):
+        if readwrite:
+            drive = f"file={self.rootfs},if=none,format=raw,id=drv0"
+        else:
+            drive = f"file={self.rootfs},if=none,readonly=on,format=raw,id=drv0"
         self.vm.add_args('-kernel', self.kernel_path,
                          '-append', self.kernel_params,
-                         '-drive',  f"file={self.rootfs},if=none,snapshot=on,format=raw,id=drv0",
+                         '-drive',  drive,
                          '-device', 'xen-disk,drive=drv0,vdev=xvda',
                          '-device', 'virtio-net-pci,netdev=unet',
                          '-netdev', 'user,id=unet,hostfwd=:127.0.0.1:0-:22')
@@ -90,11 +99,11 @@ def test_kvm_xen_guest(self):
         :avocado: tags=kvm_xen_guest
         """
 
-        self.common_vm_setup()
+        self.common_vm_setup(True)
 
         self.kernel_params = (self.KERNEL_DEFAULT +
                               ' xen_emul_unplug=ide-disks')
-        self.run_and_check()
+        self.run_and_check(True)
         self.ssh_command('grep xen-pirq.*msi /proc/interrupts')
 
     def test_kvm_xen_guest_nomsi(self):
@@ -102,11 +111,11 @@ def test_kvm_xen_guest_nomsi(self):
         :avocado: tags=kvm_xen_guest_nomsi
         """
 
-        self.common_vm_setup()
+        self.common_vm_setup(True)
 
         self.kernel_params = (self.KERNEL_DEFAULT +
                               ' xen_emul_unplug=ide-disks pci=nomsi')
-        self.run_and_check()
+        self.run_and_check(True)
         self.ssh_command('grep xen-pirq.* /proc/interrupts')
 
     def test_kvm_xen_guest_noapic_nomsi(self):
@@ -114,11 +123,11 @@ def test_kvm_xen_guest_noapic_nomsi(self):
         :avocado: tags=kvm_xen_guest_noapic_nomsi
         """
 
-        self.common_vm_setup()
+        self.common_vm_setup(True)
 
         self.kernel_params = (self.KERNEL_DEFAULT +
                               ' xen_emul_unplug=ide-disks noapic pci=nomsi')
-        self.run_and_check()
+        self.run_and_check(True)
         self.ssh_command('grep xen-pirq /proc/interrupts')
 
     def test_kvm_xen_guest_vapic(self):
-- 
2.45.2

[PATCH 08/13] testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset

[Cleber Rosa]

The asset used in the mentioned test gets truncated before it's used
in the test.  This means that the file gets modified, and thus the
asset's expected hash doesn't match anymore.  This causes cache misses
and re-downloads every time the test is re-run.

Let's make a copy of the asset so that the one in the cache is
preserved and the cache sees a hit on re-runs.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/boot_linux_console.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
index b8b0a4df10..2929aa042d 100644
--- a/tests/avocado/boot_linux_console.py
+++ b/tests/avocado/boot_linux_console.py
@@ -401,14 +401,16 @@ def test_arm_emcraft_sf2(self):
                    'fe371d32e50ca682391e1e70ab98c2942aeffb01/spi.bin')
         spi_hash = '65523a1835949b6f4553be96dec1b6a38fb05501'
         spi_path = self.fetch_asset(spi_url, asset_hash=spi_hash)
+        spi_path_rw = os.path.join(self.workdir, os.path.basename(spi_path))
+        shutil.copy(spi_path, spi_path_rw)
 
-        file_truncate(spi_path, 16 << 20) # Spansion S25FL128SDPBHICO is 16 MiB
+        file_truncate(spi_path_rw, 16 << 20) # Spansion S25FL128SDPBHICO is 16 MiB
 
         self.vm.set_console()
         kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE
         self.vm.add_args('-kernel', uboot_path,
                          '-append', kernel_command_line,
-                         '-drive', 'file=' + spi_path + ',if=mtd,format=raw',
+                         '-drive', 'file=' + spi_path_rw + ',if=mtd,format=raw',
                          '-no-reboot')
         self.vm.launch()
         self.wait_for_console_pattern('Enter \'help\' for a list')
-- 
2.45.2

[PATCH 09/13] tests/avocado/boot_xen.py: fetch kernel during test setUp()

[Cleber Rosa]

The kernel is a common blob used in all tests.  By moving it to the
setUp() method, the "fetch asset" plugin will recognize the kernel and
attempt to fetch it and cache it before the tests are started.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/boot_xen.py | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/tests/avocado/boot_xen.py b/tests/avocado/boot_xen.py
index f29bc58b9e..490a127a3e 100644
--- a/tests/avocado/boot_xen.py
+++ b/tests/avocado/boot_xen.py
@@ -30,23 +30,22 @@ class BootXen(LinuxKernelTest):
     timeout = 90
     XEN_COMMON_COMMAND_LINE = 'dom0_mem=128M loglvl=all guest_loglvl=all'
 
-    def fetch_guest_kernel(self):
+    def setUp(self):
+        super(BootXen, self).setUp()
+
         # Using my own built kernel - which works
         kernel_url = ('https://fileserver.linaro.org/'
                       's/JSsewXGZ6mqxPr5/download?path=%2F&files='
                       'linux-5.9.9-arm64-ajb')
         kernel_sha1 = '4f92bc4b9f88d5ab792fa7a43a68555d344e1b83'
-        kernel_path = self.fetch_asset(kernel_url,
-                                       asset_hash=kernel_sha1)
-
-        return kernel_path
+        self.kernel_path = self.fetch_asset(kernel_url,
+                                            asset_hash=kernel_sha1)
 
     def launch_xen(self, xen_path):
         """
         Launch Xen with a dom0 guest kernel
         """
         self.log.info("launch with xen_path: %s", xen_path)
-        kernel_path = self.fetch_guest_kernel()
 
         self.vm.set_console()
 
@@ -56,7 +55,7 @@ def launch_xen(self, xen_path):
                          '-append', self.XEN_COMMON_COMMAND_LINE,
                          '-device',
                          'guest-loader,addr=0x47000000,kernel=%s,bootargs=console=hvc0'
-                         % (kernel_path))
+                         % (self.kernel_path))
 
         self.vm.launch()
 
-- 
2.45.2

[PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

[Cleber Rosa]

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/tuxrun_baselines.py | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)

diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py
index 736e4aa289..bd02e88ed6 100644
--- a/tests/avocado/tuxrun_baselines.py
+++ b/tests/avocado/tuxrun_baselines.py
@@ -17,6 +17,7 @@
 from avocado_qemu import QemuSystemTest
 from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
 from avocado_qemu import wait_for_console_pattern
+from avocado.utils import archive
 from avocado.utils import process
 from avocado.utils.path import find_command
 
@@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None):
 
         return default
 
+    @skipUnless(archive._probe_zstd_cmd(),
+                'Could not find "zstd", or it is not able to properly '
+                'decompress decompress the rootfs')
     def setUp(self):
         super().setUp()
 
-        # We need zstd for all the tuxrun tests
-        # See https://github.com/avocado-framework/avocado/issues/5609
-        zstd = find_command('zstd', False)
-        if zstd is False:
-            self.cancel('Could not find "zstd", which is required to '
-                        'decompress rootfs')
-        self.zstd = zstd
-
         # Process the TuxRun specific tags, most machines work with
         # reasonable defaults but we sometimes need to tweak the
         # config. To avoid open coding everything we store all these
@@ -99,8 +95,8 @@ def fetch_tuxrun_assets(self, csums=None, dt=None):
                                          asset_hash = isum,
                                          algorithm = "sha256")
 
-        cmd = f"{self.zstd} -d {disk_image_zst} -o {self.workdir}/rootfs.ext4"
-        process.run(cmd)
+        archive.extract(disk_image_zst, os.path.join(self.workdir,
+                                                     "rootfs.ext4"))
 
         if dt:
             dsum = csums.get(dt, None)
-- 
2.45.2

[PATCH 11/13] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image

[Cleber Rosa]

When the OpenBSD based tests are run in parallel, the previously
single instance of the image would become corrupt.  Let's give each
test its own copy.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/machine_aarch64_sbsaref.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
index 1275f24532..8816308b86 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -7,6 +7,7 @@
 # SPDX-License-Identifier: GPL-2.0-or-later
 
 import os
+import shutil
 
 from avocado import skipUnless
 from avocado.utils import archive
@@ -187,7 +188,9 @@ def boot_openbsd73(self, cpu):
         )
 
         img_hash = "7fc2c75401d6f01fbfa25f4953f72ad7d7c18650056d30755c44b9c129b707e5"
-        img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash)
+        cached_img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash)
+        img_path = os.path.join(self.workdir, os.path.basename(cached_img_path))
+        shutil.copy(cached_img_path, img_path)
 
         self.vm.set_console()
         self.vm.add_args(
-- 
2.45.2

[PATCH 12/13] Bump avocado to 103.0

[Cleber Rosa]

This bumps Avocado to latest the LTS release.

An LTS release is one that can receive bugfixes and guarantees
stability for a much longer period and has incremental minor releases
made.

Even though the 103.0 LTS release is pretty a rewrite of Avocado when
compared to 88.1, the behavior of all existing tests under
tests/avocado has been extensively tested no regression in behavior
was found.

Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 pythondeps.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pythondeps.toml b/pythondeps.toml
index f6e590fdd8..175cf99241 100644
--- a/pythondeps.toml
+++ b/pythondeps.toml
@@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 # Note that qemu.git/python/ is always implicitly installed.
 # Prefer an LTS version when updating the accepted versions of
 # avocado-framework, for example right now the limit is 92.x.
-avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
+avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" }
 pycdlib = { accepted = ">=1.11.0" }
-- 
2.45.2

[PATCH 13/13] Avocado tests: allow for parallel execution of tests

[Cleber Rosa]

The updated Avocado version allows for the execution of tests in
parallel.

While on a CI environment it may not be a good idea to increase the
parallelization level in a single runner, developers may leverage that
on specific CI runners or on their development environments.

This also multiplies the timeout for each test accordingly.  The
reason is that more concurrency can lead to less resources, and less
resources can lead to some specific tests taking longer to complete
and then time out.  The timeout factor being used here is very
conservative (being equal to the amount of parallel tasks).  The worst
this possibly oversized timeout value can do is making users wait a
bit longer for the job to finish if a test hangs.

Overall, users can expect a much quicker turnaround on most systems
with a value such as 8 on a 12 core machine.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 docs/devel/testing.rst | 12 ++++++++++++
 tests/Makefile.include |  6 +++++-
 2 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 23d3f44f52..5600123743 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -983,6 +983,18 @@ of Avocado or ``make check-avocado``, and can also be queried using:
 
   pyvenv/bin/avocado list tests/avocado
 
+To run tests in parallel, the ``AVOCADO_PARALLEL`` environment
+variable can be defined with a value different than ``1`` (its default
+value).  Example:
+
+ .. code::
+
+  make check-avocado AVOCADO_PARALLEL=4
+
+Please exercise care when using parallel execution with the QEMU
+Avocado tests as a higher system load can cause time sensitive tests
+to timeout and be interrupted.
+
 Manual Installation
 ~~~~~~~~~~~~~~~~~~~
 
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 6618bfed70..545b5155f9 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 ifndef AVOCADO_TESTS
 	AVOCADO_TESTS=tests/avocado
 endif
+ifndef AVOCADO_PARALLEL
+	AVOCADO_PARALLEL=1
+endif
 # Controls the output generated by Avocado when running tests.
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
@@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
             $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
 			--filter-by-tags-include-empty-key) \
-            $(AVOCADO_CMDLINE_TAGS) \
+            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \
+			-p timeout_factor=$(AVOCADO_PARALLEL) \
             $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
             "AVOCADO", "tests/avocado")
 
-- 
2.45.2

Re: [PATCH 00/13] Bump Avocado to 103.0 LTS and update tests for compatibility and new features

[Akihiko Odaki]

On 2024/07/26 22:44, Cleber Rosa wrote:
> This is a *long* overdue update of the Avocado version used in QEMU.
> It comes a time where the role of the runner and the libraries are
> being discussed and questioned.
> 
> These exact commits have been staging on my side for over 30 days now,
> and I was exceeding what I should in terms of testing before posting.
> I apologize for the miscalculation.
> 
> Nevertheless, as pointed out, on the ML, these changes are needed NOW.
> 
> Some examples of runs in the CI can be seen below:
> 
> * Serial with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7074346143#L220):
>     RESULTS    : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
>     JOB TIME   : 432.63 s
> 
> * Parallel with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478#L222)
>     RESULTS    : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
>     JOB TIME   : 148.99 s
> 
> Cleber Rosa (13):
>    tests/avocado: mips: fallback to HTTP given certificate expiration
>    tests/avocado: mips: add hint for fetchasset plugin
>    tests/avocado/intel_iommu.py: increase timeout
>    tests/avocado: add cdrom permission related tests
>    tests/avocado: machine aarch64: standardize location and RO access
>    tests/avocado: use more distinct names for assets
>    tests/avocado/kvm_xen_guest.py: cope with asset RW requirements
>    testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset
>    tests/avocado/boot_xen.py: fetch kernel during test setUp()
>    tests/avocado/tuxrun_baselines.py: use Avocado's zstd support
>    tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
>    Bump avocado to 103.0
>    Avocado tests: allow for parallel execution of tests
> 
>   docs/devel/testing.rst                   | 12 +++++++
>   pythondeps.toml                          |  2 +-
>   tests/Makefile.include                   |  6 +++-
>   tests/avocado/boot_linux_console.py      | 24 ++++++++------
>   tests/avocado/boot_xen.py                | 13 ++++----
>   tests/avocado/cdrom.py                   | 41 ++++++++++++++++++++++++
>   tests/avocado/intel_iommu.py             |  2 ++
>   tests/avocado/kvm_xen_guest.py           | 30 +++++++++++------
>   tests/avocado/machine_aarch64_sbsaref.py | 11 +++++--
>   tests/avocado/machine_aarch64_virt.py    | 14 ++++----
>   tests/avocado/netdev-ethtool.py          |  3 +-
>   tests/avocado/tuxrun_baselines.py        | 16 ++++-----
>   12 files changed, 125 insertions(+), 49 deletions(-)
>   create mode 100644 tests/avocado/cdrom.py
> 

Reviewed-by: Akihiko Odaki <akihiko.odaki@daynix.com>

Re: [PATCH 03/13] tests/avocado/intel_iommu.py: increase timeout

[Daniel P. Berrangé]

On Fri, Jul 26, 2024 at 09:44:28AM -0400, Cleber Rosa wrote:
> Based on many runs, the average run time for these 4 tests is around
> 250 seconds, with 320 seconds being the ceiling.  In any way, the
> default 120 seconds timeout is inappropriate in my experience.
> Let's increase the timeout so these tests get a chance to completion.

A high watermark of over 5 minutes is pretty long for a test.

Looking at the test I see it runs

   self.ssh_command('dnf -y install numactl-devel')

but then never actually uses the installed package.

I expect that most of the wallclock time here is coming from having
dnf download all the repodata, 4 times over.

If the intention was to test networking, then replace this with
something that doesn't have to download 100's of MB of data, then
see what kind of running time we get before increasing any timeout. 


> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/avocado/intel_iommu.py | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/tests/avocado/intel_iommu.py b/tests/avocado/intel_iommu.py
> index 008f214397..9e7965c5df 100644
> --- a/tests/avocado/intel_iommu.py
> +++ b/tests/avocado/intel_iommu.py
> @@ -25,6 +25,8 @@ class IntelIOMMU(LinuxTest):
>      :avocado: tags=flaky
>      """
>  
> +    timeout = 360
> +
>      IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'
>      kernel_path = None
>      initrd_path = None
> -- 
> 2.45.2
> 
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 05/13] tests/avocado: machine aarch64: standardize location and RO access

[Daniel P. Berrangé]

On Fri, Jul 26, 2024 at 09:44:30AM -0400, Cleber Rosa wrote:
> The tests under machine_aarch64_virt.py and machine_aarch64_sbsaref.py
> should not be writing to the ISO files.  By adding "media=cdrom" the
> "ro" is autmatically set.
> 
> While at it, let's use a single code style and hash for the ISO url.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/avocado/machine_aarch64_sbsaref.py |  6 +++++-
>  tests/avocado/machine_aarch64_virt.py    | 14 +++++++-------
>  2 files changed, 12 insertions(+), 8 deletions(-)
> 
> diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
> index e920bbf08c..1275f24532 100644
> --- a/tests/avocado/machine_aarch64_sbsaref.py
> +++ b/tests/avocado/machine_aarch64_sbsaref.py
> @@ -129,7 +129,11 @@ def boot_alpine_linux(self, cpu):
>              "-cpu",
>              cpu,
>              "-drive",
> -            f"file={iso_path},format=raw",
> +            f"file={iso_path},media=cdrom,format=raw",
> +            "-device",
> +            "virtio-rng-pci,rng=rng0",
> +            "-object",
> +            "rng-random,id=rng0,filename=/dev/urandom",
>          )

The commit message doesn't say anything about adding virtio-rng.
If that's needed for some reason, do it as a separate commit
with an explanation of the bug its fixing.


> diff --git a/tests/avocado/machine_aarch64_virt.py b/tests/avocado/machine_aarch64_virt.py
> index a90dc6ff4b..6831d2c0ed 100644
> --- a/tests/avocado/machine_aarch64_virt.py
> +++ b/tests/avocado/machine_aarch64_virt.py
> @@ -37,13 +37,13 @@ def test_alpine_virt_tcg_gic_max(self):
>          :avocado: tags=machine:virt
>          :avocado: tags=accel:tcg
>          """
> -        iso_url = ('https://dl-cdn.alpinelinux.org/'
> -                   'alpine/v3.17/releases/aarch64/'
> -                   'alpine-standard-3.17.2-aarch64.iso')
> +        iso_url = (
> +            "https://dl-cdn.alpinelinux.org/"
> +            "alpine/v3.17/releases/aarch64/alpine-standard-3.17.2-aarch64.iso"
> +        )
>  
> -        # Alpine use sha256 so I recalculated this myself
> -        iso_sha1 = '76284fcd7b41fe899b0c2375ceb8470803eea839'
> -        iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha1)
> +        iso_hash = "5a36304ecf039292082d92b48152a9ec21009d3a62f459de623e19c4bd9dc027"
> +        iso_path = self.fetch_asset(iso_url, algorithm="sha256", asset_hash=iso_hash)
>  
>          self.vm.set_console()
>          kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
> @@ -60,7 +60,7 @@ def test_alpine_virt_tcg_gic_max(self):
>          self.vm.add_args("-smp", "2", "-m", "1024")
>          self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios',
>                                                 'edk2-aarch64-code.fd'))
> -        self.vm.add_args("-drive", f"file={iso_path},format=raw")
> +        self.vm.add_args("-drive", f"file={iso_path},media=cdrom,format=raw")
>          self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
>          self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
>  
> -- 
> 2.45.2
> 
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 06/13] tests/avocado: use more distinct names for assets

[Daniel P. Berrangé]

On Fri, Jul 26, 2024 at 09:44:31AM -0400, Cleber Rosa wrote:
> Avocado's asset system will deposit files in a cache organized either
> by their original location (the URI) or by their names.  Because the
> cache (and the "by_name" sub directory) is common across tests, it's a
> good idea to make these names as distinct as possible.
> 
> This avoid name clashes, which makes future Avocado runs to attempt to
> redownload the assets with the same name, but from the different
> locations they actually are from.  This causes cache misses, extra
> downloads, and possibly canceled tests.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/avocado/kvm_xen_guest.py  | 3 ++-
>  tests/avocado/netdev-ethtool.py | 3 ++-
>  2 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/avocado/kvm_xen_guest.py b/tests/avocado/kvm_xen_guest.py
> index f8cb458d5d..318fadebc3 100644
> --- a/tests/avocado/kvm_xen_guest.py
> +++ b/tests/avocado/kvm_xen_guest.py
> @@ -40,7 +40,8 @@ def get_asset(self, name, sha1):
>          url = base_url + name
>          # use explicit name rather than failing to neatly parse the
>          # URL into a unique one
> -        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
> +        return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
> +                                locations=(url), asset_hash=sha1)

Why do we need to pass a name here at all ? I see the comment here
but it isn't very clear about what the problem is. It just feels
wrong to be creating ourselves uniqueness naming problems, when we
have a nicely unique URL, and that cached URL can be shared across
tests, where as the custom names added by this patch are forcing
no-caching of the same URL between tests.

>  
>      def common_vm_setup(self):
>          # We also catch lack of KVM_XEN support if we fail to launch
> diff --git a/tests/avocado/netdev-ethtool.py b/tests/avocado/netdev-ethtool.py
> index 5f33288f81..462cf8de7d 100644
> --- a/tests/avocado/netdev-ethtool.py
> +++ b/tests/avocado/netdev-ethtool.py
> @@ -27,7 +27,8 @@ def get_asset(self, name, sha1):
>          url = base_url + name
>          # use explicit name rather than failing to neatly parse the
>          # URL into a unique one
> -        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
> +        return self.fetch_asset(name=f"qemu-netdev-ethtool-{name}",
> +                                locations=(url), asset_hash=sha1)
>  
>      def common_test_code(self, netdev, extra_args=None):
>  
> -- 
> 2.45.2
> 
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 07/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements

[Daniel P. Berrangé]

On Fri, Jul 26, 2024 at 09:44:32AM -0400, Cleber Rosa wrote:
> Some of these tests actually require the root filesystem image,
> obtained through Avocado's asset feature and kept in a common cache
> location, to be writable.
> 
> This makes a distinction between the tests that actually have this
> requirement and those who don't.  The goal is to be as safe as
> possible, avoiding causing cache misses (because the assets get
> modified and thus need to be dowloaded again) while avoid copying the
> root filesystem backing file whenever possible.
> 
> This also allow these tests to be run in parallel with newer Avocado
> versions.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/avocado/kvm_xen_guest.py | 27 ++++++++++++++++++---------
>  1 file changed, 18 insertions(+), 9 deletions(-)
> 
> diff --git a/tests/avocado/kvm_xen_guest.py b/tests/avocado/kvm_xen_guest.py
> index 318fadebc3..d73fa888ef 100644
> --- a/tests/avocado/kvm_xen_guest.py
> +++ b/tests/avocado/kvm_xen_guest.py
> @@ -10,6 +10,7 @@
>  # SPDX-License-Identifier: GPL-2.0-or-later
>  
>  import os
> +import shutil
>  
>  from qemu.machine import machine
>  
> @@ -43,7 +44,7 @@ def get_asset(self, name, sha1):
>          return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
>                                  locations=(url), asset_hash=sha1)
>  
> -    def common_vm_setup(self):
> +    def common_vm_setup(self, readwrite=False):
>          # We also catch lack of KVM_XEN support if we fail to launch
>          self.require_accelerator("kvm")
>  
> @@ -56,11 +57,19 @@ def common_vm_setup(self):
>                                            "367962983d0d32109998a70b45dcee4672d0b045")
>          self.rootfs = self.get_asset("rootfs.ext4",
>                                       "f1478401ea4b3fa2ea196396be44315bab2bb5e4")
> +        if readwrite:
> +            dest = os.path.join(self.workdir, os.path.basename(self.rootfs))
> +            shutil.copy(self.rootfs, dest)
> +            self.rootfs = dest

This is a very expensive way of creating a writable disk. Better to
avoid adding this 'readwrite' parameter at all, and instead create
a throwaway qcow2 overlay for the image for all tests. That ensures
writability for everything in a cheap manner.

>  
> -    def run_and_check(self):
> +    def run_and_check(self, readwrite=False):
> +        if readwrite:
> +            drive = f"file={self.rootfs},if=none,format=raw,id=drv0"
> +        else:
> +            drive = f"file={self.rootfs},if=none,readonly=on,format=raw,id=drv0"
>          self.vm.add_args('-kernel', self.kernel_path,
>                           '-append', self.kernel_params,
> -                         '-drive',  f"file={self.rootfs},if=none,snapshot=on,format=raw,id=drv0",
> +                         '-drive',  drive,
>                           '-device', 'xen-disk,drive=drv0,vdev=xvda',
>                           '-device', 'virtio-net-pci,netdev=unet',
>                           '-netdev', 'user,id=unet,hostfwd=:127.0.0.1:0-:22')
> @@ -90,11 +99,11 @@ def test_kvm_xen_guest(self):
>          :avocado: tags=kvm_xen_guest
>          """
>  
> -        self.common_vm_setup()
> +        self.common_vm_setup(True)
>  
>          self.kernel_params = (self.KERNEL_DEFAULT +
>                                ' xen_emul_unplug=ide-disks')
> -        self.run_and_check()
> +        self.run_and_check(True)
>          self.ssh_command('grep xen-pirq.*msi /proc/interrupts')
>  
>      def test_kvm_xen_guest_nomsi(self):
> @@ -102,11 +111,11 @@ def test_kvm_xen_guest_nomsi(self):
>          :avocado: tags=kvm_xen_guest_nomsi
>          """
>  
> -        self.common_vm_setup()
> +        self.common_vm_setup(True)
>  
>          self.kernel_params = (self.KERNEL_DEFAULT +
>                                ' xen_emul_unplug=ide-disks pci=nomsi')
> -        self.run_and_check()
> +        self.run_and_check(True)
>          self.ssh_command('grep xen-pirq.* /proc/interrupts')
>  
>      def test_kvm_xen_guest_noapic_nomsi(self):
> @@ -114,11 +123,11 @@ def test_kvm_xen_guest_noapic_nomsi(self):
>          :avocado: tags=kvm_xen_guest_noapic_nomsi
>          """
>  
> -        self.common_vm_setup()
> +        self.common_vm_setup(True)
>  
>          self.kernel_params = (self.KERNEL_DEFAULT +
>                                ' xen_emul_unplug=ide-disks noapic pci=nomsi')
> -        self.run_and_check()
> +        self.run_and_check(True)
>          self.ssh_command('grep xen-pirq /proc/interrupts')
>  
>      def test_kvm_xen_guest_vapic(self):
> -- 
> 2.45.2
> 
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 08/13] testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset

[Daniel P. Berrangé]

On Fri, Jul 26, 2024 at 09:44:33AM -0400, Cleber Rosa wrote:
> The asset used in the mentioned test gets truncated before it's used
> in the test.  This means that the file gets modified, and thus the
> asset's expected hash doesn't match anymore.  This causes cache misses
> and re-downloads every time the test is re-run.
> 
> Let's make a copy of the asset so that the one in the cache is
> preserved and the cache sees a hit on re-runs.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/avocado/boot_linux_console.py | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 09/13] tests/avocado/boot_xen.py: fetch kernel during test setUp()

[Daniel P. Berrangé]

On Fri, Jul 26, 2024 at 09:44:34AM -0400, Cleber Rosa wrote:
> The kernel is a common blob used in all tests.  By moving it to the
> setUp() method, the "fetch asset" plugin will recognize the kernel and
> attempt to fetch it and cache it before the tests are started.

The other tests don't call  fetch_asset() from their setUp
method - what's different about this test that prevents the
asset caching working ?

> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/avocado/boot_xen.py | 13 ++++++-------
>  1 file changed, 6 insertions(+), 7 deletions(-)
> 
> diff --git a/tests/avocado/boot_xen.py b/tests/avocado/boot_xen.py
> index f29bc58b9e..490a127a3e 100644
> --- a/tests/avocado/boot_xen.py
> +++ b/tests/avocado/boot_xen.py
> @@ -30,23 +30,22 @@ class BootXen(LinuxKernelTest):
>      timeout = 90
>      XEN_COMMON_COMMAND_LINE = 'dom0_mem=128M loglvl=all guest_loglvl=all'
>  
> -    def fetch_guest_kernel(self):
> +    def setUp(self):
> +        super(BootXen, self).setUp()
> +
>          # Using my own built kernel - which works
>          kernel_url = ('https://fileserver.linaro.org/'
>                        's/JSsewXGZ6mqxPr5/download?path=%2F&files='
>                        'linux-5.9.9-arm64-ajb')
>          kernel_sha1 = '4f92bc4b9f88d5ab792fa7a43a68555d344e1b83'
> -        kernel_path = self.fetch_asset(kernel_url,
> -                                       asset_hash=kernel_sha1)
> -
> -        return kernel_path
> +        self.kernel_path = self.fetch_asset(kernel_url,
> +                                            asset_hash=kernel_sha1)
>  
>      def launch_xen(self, xen_path):
>          """
>          Launch Xen with a dom0 guest kernel
>          """
>          self.log.info("launch with xen_path: %s", xen_path)
> -        kernel_path = self.fetch_guest_kernel()
>  
>          self.vm.set_console()
>  
> @@ -56,7 +55,7 @@ def launch_xen(self, xen_path):
>                           '-append', self.XEN_COMMON_COMMAND_LINE,
>                           '-device',
>                           'guest-loader,addr=0x47000000,kernel=%s,bootargs=console=hvc0'
> -                         % (kernel_path))
> +                         % (self.kernel_path))
>  
>          self.vm.launch()
>  
> -- 
> 2.45.2
> 
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

[Daniel P. Berrangé]

On Fri, Jul 26, 2024 at 09:44:35AM -0400, Cleber Rosa wrote:
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/avocado/tuxrun_baselines.py | 16 ++++++----------
>  1 file changed, 6 insertions(+), 10 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 11/13] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image

[Daniel P. Berrangé]

On Fri, Jul 26, 2024 at 09:44:36AM -0400, Cleber Rosa wrote:
> When the OpenBSD based tests are run in parallel, the previously
> single instance of the image would become corrupt.  Let's give each
> test its own copy.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/avocado/machine_aarch64_sbsaref.py | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
> index 1275f24532..8816308b86 100644
> --- a/tests/avocado/machine_aarch64_sbsaref.py
> +++ b/tests/avocado/machine_aarch64_sbsaref.py
> @@ -7,6 +7,7 @@
>  # SPDX-License-Identifier: GPL-2.0-or-later
>  
>  import os
> +import shutil
>  
>  from avocado import skipUnless
>  from avocado.utils import archive
> @@ -187,7 +188,9 @@ def boot_openbsd73(self, cpu):
>          )
>  
>          img_hash = "7fc2c75401d6f01fbfa25f4953f72ad7d7c18650056d30755c44b9c129b707e5"
> -        img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash)
> +        cached_img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash)
> +        img_path = os.path.join(self.workdir, os.path.basename(cached_img_path))
> +        shutil.copy(cached_img_path, img_path)

Again I think we should be using a throwaway qcow2 overlay rather
than copying the full image.

>  
>          self.vm.set_console()
>          self.vm.add_args(

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 12/13] Bump avocado to 103.0

[Daniel P. Berrangé]

On Fri, Jul 26, 2024 at 09:44:37AM -0400, Cleber Rosa wrote:
> This bumps Avocado to latest the LTS release.
> 
> An LTS release is one that can receive bugfixes and guarantees
> stability for a much longer period and has incremental minor releases
> made.
> 
> Even though the 103.0 LTS release is pretty a rewrite of Avocado when
> compared to 88.1, the behavior of all existing tests under
> tests/avocado has been extensively tested no regression in behavior
> was found.

Rebasing to a completely re-written test harness while in freeze feels
on the risky side to me, despite the known problems we have with the
existing release. 

> Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  pythondeps.toml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/pythondeps.toml b/pythondeps.toml
> index f6e590fdd8..175cf99241 100644
> --- a/pythondeps.toml
> +++ b/pythondeps.toml
> @@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
>  # Note that qemu.git/python/ is always implicitly installed.
>  # Prefer an LTS version when updating the accepted versions of
>  # avocado-framework, for example right now the limit is 92.x.
> -avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
> +avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" }
>  pycdlib = { accepted = ">=1.11.0" }
> -- 
> 2.45.2
> 
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 01/13] tests/avocado: mips: fallback to HTTP given certificate expiration

[Philippe Mathieu-Daudé]

On 26/7/24 15:44, Cleber Rosa wrote:
> The SSL certificate installed at mipsdistros.mips.com has expired:
> 
>   0 s:CN = mipsdistros.mips.com
>   i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
>   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
>   v:NotBefore: Dec 23 00:00:00 2019 GMT; NotAfter: Jan 23 12:00:00 2021 GMT
> 
> Because this project has no control over that certificate and host,
> this falls back to plain HTTP instead.  The integrity of the
> downloaded files can be guaranteed by the existing hashes for those
> files (which are not modified here).
> 
> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   tests/avocado/boot_linux_console.py | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)

Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH 05/13] tests/avocado: machine aarch64: standardize location and RO access

[Cleber Rosa]

On Mon, Jul 29, 2024 at 6:34 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Fri, Jul 26, 2024 at 09:44:30AM -0400, Cleber Rosa wrote:
> > The tests under machine_aarch64_virt.py and machine_aarch64_sbsaref.py
> > should not be writing to the ISO files.  By adding "media=cdrom" the
> > "ro" is autmatically set.
> >
> > While at it, let's use a single code style and hash for the ISO url.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  tests/avocado/machine_aarch64_sbsaref.py |  6 +++++-
> >  tests/avocado/machine_aarch64_virt.py    | 14 +++++++-------
> >  2 files changed, 12 insertions(+), 8 deletions(-)
> >
> > diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
> > index e920bbf08c..1275f24532 100644
> > --- a/tests/avocado/machine_aarch64_sbsaref.py
> > +++ b/tests/avocado/machine_aarch64_sbsaref.py
> > @@ -129,7 +129,11 @@ def boot_alpine_linux(self, cpu):
> >              "-cpu",
> >              cpu,
> >              "-drive",
> > -            f"file={iso_path},format=raw",
> > +            f"file={iso_path},media=cdrom,format=raw",
> > +            "-device",
> > +            "virtio-rng-pci,rng=rng0",
> > +            "-object",
> > +            "rng-random,id=rng0,filename=/dev/urandom",
> >          )
>
> The commit message doesn't say anything about adding virtio-rng.
> If that's needed for some reason, do it as a separate commit
> with an explanation of the bug its fixing.
>

This is actually a rebase mistake.  virtio-rng was removed in 21f123f3c.

I'll fix it in a v2.

Thanks for spotting it,
- Cleber.

Re: [PATCH 06/13] tests/avocado: use more distinct names for assets

[Philippe Mathieu-Daudé]

On 29/7/24 12:49, Daniel P. Berrangé wrote:
> On Fri, Jul 26, 2024 at 09:44:31AM -0400, Cleber Rosa wrote:
>> Avocado's asset system will deposit files in a cache organized either
>> by their original location (the URI) or by their names.  Because the
>> cache (and the "by_name" sub directory) is common across tests, it's a
>> good idea to make these names as distinct as possible.
>>
>> This avoid name clashes, which makes future Avocado runs to attempt to
>> redownload the assets with the same name, but from the different
>> locations they actually are from.  This causes cache misses, extra
>> downloads, and possibly canceled tests.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   tests/avocado/kvm_xen_guest.py  | 3 ++-
>>   tests/avocado/netdev-ethtool.py | 3 ++-
>>   2 files changed, 4 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/avocado/kvm_xen_guest.py b/tests/avocado/kvm_xen_guest.py
>> index f8cb458d5d..318fadebc3 100644
>> --- a/tests/avocado/kvm_xen_guest.py
>> +++ b/tests/avocado/kvm_xen_guest.py
>> @@ -40,7 +40,8 @@ def get_asset(self, name, sha1):
>>           url = base_url + name
>>           # use explicit name rather than failing to neatly parse the
>>           # URL into a unique one
>> -        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
>> +        return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
>> +                                locations=(url), asset_hash=sha1)
> 
> Why do we need to pass a name here at all ? I see the comment here
> but it isn't very clear about what the problem is. It just feels
> wrong to be creating ourselves uniqueness naming problems, when we
> have a nicely unique URL, and that cached URL can be shared across
> tests, where as the custom names added by this patch are forcing
> no-caching of the same URL between tests.

I thought $name was purely for debugging; the file was downloaded
in a temporary location, and if the hash matched, it was renamed
in the cache as $asset_hash which is unique. This was suggested
in order to avoid dealing with URL updates for the same asset.
Isn't it the case?

Re: [PATCH 12/13] Bump avocado to 103.0

[Philippe Mathieu-Daudé]

On 26/7/24 15:44, Cleber Rosa wrote:
> This bumps Avocado to latest the LTS release.
> 
> An LTS release is one that can receive bugfixes and guarantees
> stability for a much longer period and has incremental minor releases
> made.
> 
> Even though the 103.0 LTS release is pretty a rewrite of Avocado when
> compared to 88.1, the behavior of all existing tests under
> tests/avocado has been extensively tested no regression in behavior
> was found.

Does that restore feature parity for macOS developers? Because this
community has been left behind ignored for over 2 years and already
looked at alternatives for functional testing.

> Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   pythondeps.toml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/pythondeps.toml b/pythondeps.toml
> index f6e590fdd8..175cf99241 100644
> --- a/pythondeps.toml
> +++ b/pythondeps.toml
> @@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
>   # Note that qemu.git/python/ is always implicitly installed.
>   # Prefer an LTS version when updating the accepted versions of
>   # avocado-framework, for example right now the limit is 92.x.
> -avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
> +avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" }
>   pycdlib = { accepted = ">=1.11.0" }

Re: [PATCH 07/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1340 bytes --]

On Mon, 2024-07-29 at 11:58 +0100, Daniel P. Berrangé wrote:
> On Fri, Jul 26, 2024 at 09:44:32AM -0400, Cleber Rosa wrote:
> > Some of these tests actually require the root filesystem image,
> > obtained through Avocado's asset feature and kept in a common cache
> > location, to be writable.

Hm, I'm not sure *why* they require a writable image. Mostly they're
just testing the interrupt routing. What's the failure mode for a read-
only image?

> > @@ -56,11 +57,19 @@ def common_vm_setup(self):
> >                                            "367962983d0d32109998a70b45dcee4672d0b045")
> >          self.rootfs = self.get_asset("rootfs.ext4",
> >                                       "f1478401ea4b3fa2ea196396be44315bab2bb5e4")
> > +        if readwrite:
> > +            dest = os.path.join(self.workdir, os.path.basename(self.rootfs))
> > +            shutil.copy(self.rootfs, dest)
> > +            self.rootfs = dest
> 
> This is a very expensive way of creating a writable disk. Better to
> avoid adding this 'readwrite' parameter at all, and instead create
> a throwaway qcow2 overlay for the image for all tests. That ensures
> writability for everything in a cheap manner.

Or just use -snapshot?


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5965 bytes --]

Re: [PATCH 02/13] tests/avocado: mips: add hint for fetchasset plugin

[Philippe Mathieu-Daudé]

On 26/7/24 15:44, Cleber Rosa wrote:
> Avocado's fetchasset plugin runs before the actual Avocado job (and
> any test).  It analyses the test's code looking for occurrences of
> "self.fetch_asset()" in the either the actual test or setUp() method.
> It's not able to fully analyze all code, though.
> 
> The way these tests are written, make the fetchasset plugin blind to
> the assets.  This adds some more code duplication, true, but it will
> aid the fetchasset plugin to download or verify the existence of these
> assets in advance.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   tests/avocado/boot_linux_console.py | 12 +++++++-----
>   1 file changed, 7 insertions(+), 5 deletions(-)

Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

[Philippe Mathieu-Daudé]

On 26/7/24 15:44, Cleber Rosa wrote:
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   tests/avocado/tuxrun_baselines.py | 16 ++++++----------
>   1 file changed, 6 insertions(+), 10 deletions(-)
> 
> diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py
> index 736e4aa289..bd02e88ed6 100644
> --- a/tests/avocado/tuxrun_baselines.py
> +++ b/tests/avocado/tuxrun_baselines.py
> @@ -17,6 +17,7 @@
>   from avocado_qemu import QemuSystemTest
>   from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
>   from avocado_qemu import wait_for_console_pattern
> +from avocado.utils import archive
>   from avocado.utils import process
>   from avocado.utils.path import find_command
>   
> @@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None):
>   
>           return default
>   
> +    @skipUnless(archive._probe_zstd_cmd(),

_probe_zstd_cmd() isn't public AFAICT, but more importantly
this doesn't work because this method has been added in v101.0.

> +                'Could not find "zstd", or it is not able to properly '
> +                'decompress decompress the rootfs')
>       def setUp(self):
>           super().setUp()
>   
> -        # We need zstd for all the tuxrun tests
> -        # See https://github.com/avocado-framework/avocado/issues/5609
> -        zstd = find_command('zstd', False)
> -        if zstd is False:
> -            self.cancel('Could not find "zstd", which is required to '
> -                        'decompress rootfs')
> -        self.zstd = zstd
> -
>           # Process the TuxRun specific tags, most machines work with
>           # reasonable defaults but we sometimes need to tweak the
>           # config. To avoid open coding everything we store all these
> @@ -99,8 +95,8 @@ def fetch_tuxrun_assets(self, csums=None, dt=None):
>                                            asset_hash = isum,
>                                            algorithm = "sha256")
>   
> -        cmd = f"{self.zstd} -d {disk_image_zst} -o {self.workdir}/rootfs.ext4"
> -        process.run(cmd)
> +        archive.extract(disk_image_zst, os.path.join(self.workdir,
> +                                                     "rootfs.ext4"))
>   
>           if dt:
>               dsum = csums.get(dt, None)

Re: [PATCH 00/13] Bump Avocado to 103.0 LTS and update tests for compatibility and new features

[Philippe Mathieu-Daudé]

On 26/7/24 15:44, Cleber Rosa wrote:

> Cleber Rosa (13):
>    tests/avocado: mips: fallback to HTTP given certificate expiration
>    tests/avocado: mips: add hint for fetchasset plugin
>    tests/avocado/intel_iommu.py: increase timeout
>    tests/avocado: add cdrom permission related tests
>    tests/avocado: machine aarch64: standardize location and RO access
>    tests/avocado: use more distinct names for assets
>    tests/avocado/kvm_xen_guest.py: cope with asset RW requirements
>    testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset
>    tests/avocado/boot_xen.py: fetch kernel during test setUp()
>    tests/avocado/tuxrun_baselines.py: use Avocado's zstd support
>    tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
>    Bump avocado to 103.0
>    Avocado tests: allow for parallel execution of tests

Queuing patches #1, #2 and #8. Unfortunately #10 depends on Avocado
101.0 so can't be merged at this point.

Re: [PATCH 12/13] Bump avocado to 103.0

[Cleber Rosa]

On 7/29/24 8:02 AM, Philippe Mathieu-Daudé wrote:
> Does that restore feature parity for macOS developers? Because this
> community has been left behind ignored for over 2 years and already
> looked at alternatives for functional testing.
>

Hi Phillipe,


As early as Avocado 102.0,  macOS support is pretty complete. The exact 
words on the release notes[1] are:


"User of macOS will have a better experience when using Avocado. The 
full set of Avocado’s selftests are now run under macOS on CI. Please be 
advised that macOS is not currently supported at the same level of 
Linux-based operating systems due to the lack of 
contributors/maintainers with access to the needed hardware. If you are 
a user/developer and are willing to contribute to this, please let the 
Avocado team know."


When it comes to the lack of updates, that is a longer discussion 
indeed.  When it comes to alternatives, I don't expect the QEMU project 
to do anything else than what it's in its best interest. As late as this 
can be, please take it for what it's worth.  If it does any good to 
QEMU, please consider it.


Best,

- Cleber.


[1] - https://avocado-framework.readthedocs.io/en/latest/releases/102_0.html


>> Reference: 
>> https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>   pythondeps.toml | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/pythondeps.toml b/pythondeps.toml
>> index f6e590fdd8..175cf99241 100644
>> --- a/pythondeps.toml
>> +++ b/pythondeps.toml
>> @@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed 
>> = "1.1.1" }
>>   # Note that qemu.git/python/ is always implicitly installed.
>>   # Prefer an LTS version when updating the accepted versions of
>>   # avocado-framework, for example right now the limit is 92.x.
>> -avocado-framework = { accepted = "(>=88.1, <93.0)", installed = 
>> "88.1", canary = "avocado" }
>> +avocado-framework = { accepted = "(>=103.0, <104.0)", installed = 
>> "103.0", canary = "avocado" }
>>   pycdlib = { accepted = ">=1.11.0" }
>

Re: [PATCH 03/13] tests/avocado/intel_iommu.py: increase timeout

[Cleber Rosa]

On Mon, Jul 29, 2024 at 6:30 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Fri, Jul 26, 2024 at 09:44:28AM -0400, Cleber Rosa wrote:
> > Based on many runs, the average run time for these 4 tests is around
> > 250 seconds, with 320 seconds being the ceiling.  In any way, the
> > default 120 seconds timeout is inappropriate in my experience.
> > Let's increase the timeout so these tests get a chance to completion.
>
> A high watermark of over 5 minutes is pretty long for a test.
>

I agree.

> Looking at the test I see it runs
>
>    self.ssh_command('dnf -y install numactl-devel')
>
> but then never actually uses the installed package.
>
> I expect that most of the wallclock time here is coming from having
> dnf download all the repodata, 4 times over.
>

Exactly.

> If the intention was to test networking, then replace this with
> something that doesn't have to download 100's of MB of data, then
> see what kind of running time we get before increasing any timeout.
>
>

I was trying not to get in the way of the original test writer.

Eric,

Are you OK with replacing this command for a simpler file transfer?
Any suggestions?

Regards,
- Cleber.

Re: [PATCH 06/13] tests/avocado: use more distinct names for assets

[Cleber Rosa]

On Mon, Jul 29, 2024 at 6:49 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Fri, Jul 26, 2024 at 09:44:31AM -0400, Cleber Rosa wrote:
> > Avocado's asset system will deposit files in a cache organized either
> > by their original location (the URI) or by their names.  Because the
> > cache (and the "by_name" sub directory) is common across tests, it's a
> > good idea to make these names as distinct as possible.
> >
> > This avoid name clashes, which makes future Avocado runs to attempt to
> > redownload the assets with the same name, but from the different
> > locations they actually are from.  This causes cache misses, extra
> > downloads, and possibly canceled tests.
> >
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  tests/avocado/kvm_xen_guest.py  | 3 ++-
> >  tests/avocado/netdev-ethtool.py | 3 ++-
> >  2 files changed, 4 insertions(+), 2 deletions(-)
> >
> > diff --git a/tests/avocado/kvm_xen_guest.py b/tests/avocado/kvm_xen_guest.py
> > index f8cb458d5d..318fadebc3 100644
> > --- a/tests/avocado/kvm_xen_guest.py
> > +++ b/tests/avocado/kvm_xen_guest.py
> > @@ -40,7 +40,8 @@ def get_asset(self, name, sha1):
> >          url = base_url + name
> >          # use explicit name rather than failing to neatly parse the
> >          # URL into a unique one
> > -        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
> > +        return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
> > +                                locations=(url), asset_hash=sha1)
>
> Why do we need to pass a name here at all ? I see the comment here
> but it isn't very clear about what the problem is. It just feels
> wrong to be creating ourselves uniqueness naming problems, when we
> have a nicely unique URL, and that cached URL can be shared across
> tests, where as the custom names added by this patch are forcing
> no-caching of the same URL between tests.
>

Now with your comment, I do agree that this adds some unneeded
maintenance burden indeed.  Also, this was part of my pre-avocado bump
patches that would work around issues present in < 103.0.  But let me
give the complete answer.

Under 88.1 the "uniqueness" of the URL did not consider the query
parameters in the URL.  So, under 88.1:

   avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fkvm-xen-guest&files=bzImage',
...)
   avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fnetdev-ethtool&files=bzImage',
...)

Would save content to the same location:
/tmp/cache_old/by_location/2a8ecd750eb952504ad96b89576207afe1be6a8f/download.

This is no longer the case on 103.0 (actually since 92.0), the
contents of those exact assets would be saved to
'/by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fkvm-xen-guest&files=bzImage'
and /by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fnetdev-ethtool&files=bzImage'.

I personally don't like having the files named, although uniquely,
after the query parameters.  But, If this doesn't bother others more
than the maintenance burden, and Avocado version bump is applied, this
patch can be dropped.

Re: [PATCH 06/13] tests/avocado: use more distinct names for assets

[Cleber Rosa]

On Mon, Jul 29, 2024 at 7:54 AM Philippe Mathieu-Daudé
<philmd@linaro.org> wrote:
>
> On 29/7/24 12:49, Daniel P. Berrangé wrote:
> > On Fri, Jul 26, 2024 at 09:44:31AM -0400, Cleber Rosa wrote:
> >> Avocado's asset system will deposit files in a cache organized either
> >> by their original location (the URI) or by their names.  Because the
> >> cache (and the "by_name" sub directory) is common across tests, it's a
> >> good idea to make these names as distinct as possible.
> >>
> >> This avoid name clashes, which makes future Avocado runs to attempt to
> >> redownload the assets with the same name, but from the different
> >> locations they actually are from.  This causes cache misses, extra
> >> downloads, and possibly canceled tests.
> >>
> >> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> >> ---
> >>   tests/avocado/kvm_xen_guest.py  | 3 ++-
> >>   tests/avocado/netdev-ethtool.py | 3 ++-
> >>   2 files changed, 4 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/tests/avocado/kvm_xen_guest.py b/tests/avocado/kvm_xen_guest.py
> >> index f8cb458d5d..318fadebc3 100644
> >> --- a/tests/avocado/kvm_xen_guest.py
> >> +++ b/tests/avocado/kvm_xen_guest.py
> >> @@ -40,7 +40,8 @@ def get_asset(self, name, sha1):
> >>           url = base_url + name
> >>           # use explicit name rather than failing to neatly parse the
> >>           # URL into a unique one
> >> -        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
> >> +        return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
> >> +                                locations=(url), asset_hash=sha1)
> >
> > Why do we need to pass a name here at all ? I see the comment here
> > but it isn't very clear about what the problem is. It just feels
> > wrong to be creating ourselves uniqueness naming problems, when we
> > have a nicely unique URL, and that cached URL can be shared across
> > tests, where as the custom names added by this patch are forcing
> > no-caching of the same URL between tests.
>
> I thought $name was purely for debugging; the file was downloaded
> in a temporary location, and if the hash matched, it was renamed
> in the cache as $asset_hash which is unique. This was suggested
> in order to avoid dealing with URL updates for the same asset.
> Isn't it the case?
>

Hi Phillipe,

I've replied to Daniel's question, but let me repeat the relevant
parts of the $name behavior here.

---

Under 88.1 the "uniqueness" of the URL did not consider the query
parameters in the URL.  So, under 88.1:

   avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fkvm-xen-guest&files=bzImage',
...)
   avocado.utils.asset.Asset(name='bzImage',
locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fnetdev-ethtool&files=bzImage',
...)

Would save content to the same location:
/tmp/cache_old/by_location/2a8ecd750eb952504ad96b89576207afe1be6a8f/download.

This is no longer the case on 103.0 (actually since 92.0), the
contents of those exact assets would be saved to
'/by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fkvm-xen-guest&files=bzImage'
and /by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fnetdev-ethtool&files=bzImage'.

I personally don't like having the files named, although uniquely,
after the query parameters.  But, If this doesn't bother others more
than the maintenance burden, and Avocado version bump is applied, this
patch can be dropped.

---

Now, an update: instead of dropping the patch, it could be simplified
by keeping just the "name" parameter (with the URL) and dropping the
"locations" parameter (that has a single location).

Let me know if this makes sense and what you take on it.

Regards,
- Cleber.

Re: [PATCH 09/13] tests/avocado/boot_xen.py: fetch kernel during test setUp()

[Cleber Rosa]

On Mon, Jul 29, 2024 at 7:01 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Fri, Jul 26, 2024 at 09:44:34AM -0400, Cleber Rosa wrote:
> > The kernel is a common blob used in all tests.  By moving it to the
> > setUp() method, the "fetch asset" plugin will recognize the kernel and
> > attempt to fetch it and cache it before the tests are started.
>
> The other tests don't call  fetch_asset() from their setUp
> method - what's different about this test that prevents the
> asset caching working ?
>

The "wizardry" of the "fetch asset" plugin limits itself to analyzing
test methods and the setUp() method.  This test currently does that in
the "fetch_guest_kernel()" method, which makes it blind to the "fetch
asset" plugin.

If there are other tests doing that (in custom methods), then a
similar change would be desirable.

Regards,
- Cleber.

Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

[Cleber Rosa]

On Mon, Jul 29, 2024 at 10:39 AM Philippe Mathieu-Daudé
<philmd@linaro.org> wrote:
>
> On 26/7/24 15:44, Cleber Rosa wrote:
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >   tests/avocado/tuxrun_baselines.py | 16 ++++++----------
> >   1 file changed, 6 insertions(+), 10 deletions(-)
> >
> > diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py
> > index 736e4aa289..bd02e88ed6 100644
> > --- a/tests/avocado/tuxrun_baselines.py
> > +++ b/tests/avocado/tuxrun_baselines.py
> > @@ -17,6 +17,7 @@
> >   from avocado_qemu import QemuSystemTest
> >   from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
> >   from avocado_qemu import wait_for_console_pattern
> > +from avocado.utils import archive
> >   from avocado.utils import process
> >   from avocado.utils.path import find_command
> >
> > @@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None):
> >
> >           return default
> >
> > +    @skipUnless(archive._probe_zstd_cmd(),
>
> _probe_zstd_cmd() isn't public AFAICT, but more importantly
> this doesn't work because this method has been added in v101.0.
>

While it's not the best practice to use private functions, I just
couldn't accept rewriting that for the skip condition.  I can make
sure future  versions (including 103.1) make it public.

Also, these patches count on the bump to 103.0 indeed.

Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

[Philippe Mathieu-Daudé]

On 1/8/24 05:39, Cleber Rosa wrote:
> On Mon, Jul 29, 2024 at 10:39 AM Philippe Mathieu-Daudé
> <philmd@linaro.org> wrote:
>>
>> On 26/7/24 15:44, Cleber Rosa wrote:
>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>    tests/avocado/tuxrun_baselines.py | 16 ++++++----------
>>>    1 file changed, 6 insertions(+), 10 deletions(-)
>>>
>>> diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py
>>> index 736e4aa289..bd02e88ed6 100644
>>> --- a/tests/avocado/tuxrun_baselines.py
>>> +++ b/tests/avocado/tuxrun_baselines.py
>>> @@ -17,6 +17,7 @@
>>>    from avocado_qemu import QemuSystemTest
>>>    from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
>>>    from avocado_qemu import wait_for_console_pattern
>>> +from avocado.utils import archive
>>>    from avocado.utils import process
>>>    from avocado.utils.path import find_command
>>>
>>> @@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None):
>>>
>>>            return default
>>>
>>> +    @skipUnless(archive._probe_zstd_cmd(),
>>
>> _probe_zstd_cmd() isn't public AFAICT, but more importantly
>> this doesn't work because this method has been added in v101.0.
>>
> 
> While it's not the best practice to use private functions, I just
> couldn't accept rewriting that for the skip condition.  I can make
> sure future  versions (including 103.1) make it public.
> 
> Also, these patches count on the bump to 103.0 indeed.

Then either mention it in the commit description to avoid wasting
time to developers cherry-picking / testing this single patch, or
move it after the version bump, avoiding bisectability issues.

Thanks,

Phil.

Re: [PATCH 07/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements

[Alex Bennée]

David Woodhouse <dwmw2@infradead.org> writes:

> On Mon, 2024-07-29 at 11:58 +0100, Daniel P. Berrangé wrote:
>> On Fri, Jul 26, 2024 at 09:44:32AM -0400, Cleber Rosa wrote:
>> > Some of these tests actually require the root filesystem image,
>> > obtained through Avocado's asset feature and kept in a common cache
>> > location, to be writable.
>
> Hm, I'm not sure *why* they require a writable image. Mostly they're
> just testing the interrupt routing. What's the failure mode for a read-
> only image?

There are no real failures with readonly=on although you do see init
complain a bit about not being able to mount /dev and open log files.
snapshot allows that to happen but doesn't change the underlying
storage.

>
>> > @@ -56,11 +57,19 @@ def common_vm_setup(self):
>> >                                            "367962983d0d32109998a70b45dcee4672d0b045")
>> >          self.rootfs = self.get_asset("rootfs.ext4",
>> >                                       "f1478401ea4b3fa2ea196396be44315bab2bb5e4")
>> > +        if readwrite:
>> > +            dest = os.path.join(self.workdir, os.path.basename(self.rootfs))
>> > +            shutil.copy(self.rootfs, dest)
>> > +            self.rootfs = dest
>> 
>> This is a very expensive way of creating a writable disk. Better to
>> avoid adding this 'readwrite' parameter at all, and instead create
>> a throwaway qcow2 overlay for the image for all tests. That ensures
>> writability for everything in a cheap manner.
>
> Or just use -snapshot?

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 10/13] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

[Cleber Rosa]

On Thu, Aug 1, 2024 at 8:59 AM Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
> > Also, these patches count on the bump to 103.0 indeed.
>
> Then either mention it in the commit description to avoid wasting
> time to developers cherry-picking / testing this single patch, or
> move it after the version bump, avoiding bisectability issues.
>

You're right.  My bad.

Regards,
- Cleber.

> Thanks,
>
> Phil.
>

Re: [PATCH 07/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements

[Cleber Rosa]

On Thu, Aug 1, 2024 at 10:26 AM Alex Bennée <alex.bennee@linaro.org> wrote:
> There are no real failures with readonly=on although you do see init
> complain a bit about not being able to mount /dev and open log files.
> snapshot allows that to happen but doesn't change the underlying
> storage.
>

I could swear I experienced failures when I developed this patch.  But
like your patch b9371a7b90 already fixes it.

Dropping this one.

Thanks!
- Cleber.

Re: [PATCH 06/13] tests/avocado: use more distinct names for assets

[Alex Bennée]

Cleber Rosa <crosa@redhat.com> writes:

> On Mon, Jul 29, 2024 at 6:49 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>>
>> On Fri, Jul 26, 2024 at 09:44:31AM -0400, Cleber Rosa wrote:
>> > Avocado's asset system will deposit files in a cache organized either
>> > by their original location (the URI) or by their names.  Because the
>> > cache (and the "by_name" sub directory) is common across tests, it's a
>> > good idea to make these names as distinct as possible.
>> >
>> > This avoid name clashes, which makes future Avocado runs to attempt to
>> > redownload the assets with the same name, but from the different
>> > locations they actually are from.  This causes cache misses, extra
>> > downloads, and possibly canceled tests.
>> >
>> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> > ---
>> >  tests/avocado/kvm_xen_guest.py  | 3 ++-
>> >  tests/avocado/netdev-ethtool.py | 3 ++-
>> >  2 files changed, 4 insertions(+), 2 deletions(-)
>> >
>> > diff --git a/tests/avocado/kvm_xen_guest.py b/tests/avocado/kvm_xen_guest.py
>> > index f8cb458d5d..318fadebc3 100644
>> > --- a/tests/avocado/kvm_xen_guest.py
>> > +++ b/tests/avocado/kvm_xen_guest.py
>> > @@ -40,7 +40,8 @@ def get_asset(self, name, sha1):
>> >          url = base_url + name
>> >          # use explicit name rather than failing to neatly parse the
>> >          # URL into a unique one
>> > -        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
>> > +        return self.fetch_asset(name=f"qemu-kvm-xen-guest-{name}",
>> > +                                locations=(url), asset_hash=sha1)
>>
>> Why do we need to pass a name here at all ? I see the comment here
>> but it isn't very clear about what the problem is. It just feels
>> wrong to be creating ourselves uniqueness naming problems, when we
>> have a nicely unique URL, and that cached URL can be shared across
>> tests, where as the custom names added by this patch are forcing
>> no-caching of the same URL between tests.
>>
>
> Now with your comment, I do agree that this adds some unneeded
> maintenance burden indeed.  Also, this was part of my pre-avocado bump
> patches that would work around issues present in < 103.0.  But let me
> give the complete answer.
>
> Under 88.1 the "uniqueness" of the URL did not consider the query
> parameters in the URL.  So, under 88.1:
>
>    avocado.utils.asset.Asset(name='bzImage',
> locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fkvm-xen-guest&files=bzImage',
> ...)
>    avocado.utils.asset.Asset(name='bzImage',
> locations=['https://fileserver.linaro.org/s/kE4nCFLdQcoBF9t/download?path=%2Fnetdev-ethtool&files=bzImage',
> ...)

This is mostly a hack to avoid having to tell NextCloud to generate a
unique sharing URL for every file.

>
> Would save content to the same location:
> /tmp/cache_old/by_location/2a8ecd750eb952504ad96b89576207afe1be6a8f/download.
>
> This is no longer the case on 103.0 (actually since 92.0), the
> contents of those exact assets would be saved to
> '/by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fkvm-xen-guest&files=bzImage'
> and /by_location/415c998a0061347e5115da53d57ea92c908a2e7f/path=%2Fnetdev-ethtool&files=bzImage'.
>
> I personally don't like having the files named, although uniquely,
> after the query parameters.  But, If this doesn't bother others more
> than the maintenance burden, and Avocado version bump is applied, this
> patch can be dropped.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 03/13] tests/avocado/intel_iommu.py: increase timeout

[Eric Auger]

Hi Cleber,

On 8/1/24 03:02, Cleber Rosa wrote:
> On Mon, Jul 29, 2024 at 6:30 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>> On Fri, Jul 26, 2024 at 09:44:28AM -0400, Cleber Rosa wrote:
>>> Based on many runs, the average run time for these 4 tests is around
>>> 250 seconds, with 320 seconds being the ceiling.  In any way, the
>>> default 120 seconds timeout is inappropriate in my experience.
>>> Let's increase the timeout so these tests get a chance to completion.
>> A high watermark of over 5 minutes is pretty long for a test.
>>
> I agree.
>
>> Looking at the test I see it runs
>>
>>    self.ssh_command('dnf -y install numactl-devel')
>>
>> but then never actually uses the installed package.
>>
>> I expect that most of the wallclock time here is coming from having
>> dnf download all the repodata, 4 times over.
>>
> Exactly.
>
>> If the intention was to test networking, then replace this with
>> something that doesn't have to download 100's of MB of data, then
>> see what kind of running time we get before increasing any timeout.
>>
>>
> I was trying not to get in the way of the original test writer.
>
> Eric,
>
> Are you OK with replacing this command for a simpler file transfer?
> Any suggestions?
Sorry I am just returning from PTO. Yes I am OK to replace it with a
file transfer if it takes too much time. When the test was written we
just found out a bug with that exact dnf install command so at that time
this was a reproducer. Either replace it with a dummy file transfer or
withdraw it from the automated tests.

Thanks

Eric
>
> Regards,
> - Cleber.
>