[PATCH v2 0/9] Bump Avocado to 103.0 LTS and update tests for compatibility and new features

[PATCH v2 0/9] Bump Avocado to 103.0 LTS and update tests for compatibility and new features

[Cleber Rosa]

This is a *long* overdue update of the Avocado version used in QEMU.
It comes a time where the role of the runner and the libraries are
being discussed and questioned.

These exact commits have been staging on my side for over 30 days now,
and I was exceeding what I should in terms of testing before posting.
I apologize for the miscalculation.

Nevertheless, as pointed out, on the ML, these changes are needed NOW.

Some examples of runs in the CI can be seen below:

* Serial with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7074346143#L220):
   RESULTS    : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
   JOB TIME   : 432.63 s

* Parallel with 103.0 LTS (https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478#L222)
   RESULTS    : PASS 46 | ERROR 0 | FAIL 0 | SKIP 2 | WARN 0 | INTERRUPT 0 | CANCEL 0
   JOB TIME   : 148.99 s

Changes from v1 (references are from v1):

This version first applies the Avocado version bump (now PATCH 1),
adding to that patch the serial execution behavior.  Avocado 103.0 LTS
can run all existing tests in QEMU without any changes, so it's safe
to pick PATCH 1 *only* from this series if needed.  A GitLab CI job that
does only that can be seen here:

 https://gitlab.com/cleber.gnu/qemu/-/pipelines/1402633650

Details on changes:

 * Moved "Bump avocado to 103.0" to first patch

 * Patches already applied:
    - [PATCH 1/13] tests/avocado: mips: fallback to HTTP given certificate expiration
    - [PATCH 2/13] tests/avocado: mips: add hint for fetchasset plugin
    - [PATCH 8/13] testa/avocado: test_arm_emcraft_sf2: handle RW requirements for asset

  * Dropped patches:
    - [PATCH 3/13] tests/avocado/intel_iommu.py: increase timeout
    - [PATCH 6/13] tests/avocado: use more distinct names for assets
      - Replaced with "tests/avocado: simplify parameters on fetch_asset with name only"
    - [PATCH 7/13] tests/avocado/kvm_xen_guest.py: cope with asset RW requirements

  * [PATCH 5/13] tests/avocado: machine aarch64: standardize location and RO access:
    - Fixed rebase mistake

  * [PATCH 11/13] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
    - Use "snapshot=on" instead of new copy of file

  * New patches:
    - tests/avocado: apply proper skipUnless decorator
    - tests/avocado: simplify parameters on fetch_asset with name only

Cleber Rosa (9):
  Bump avocado to 103.0
  tests/avocado: apply proper skipUnless decorator
  tests/avocado: add cdrom permission related tests
  tests/avocado: machine aarch64: standardize location and RO access
  tests/avocado: simplify parameters on fetch_asset with name only
  tests/avocado/boot_xen.py: fetch kernel during test setUp()
  tests/avocado/tuxrun_baselines.py: use Avocado's zstd support
  tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
  Avocado tests: allow for parallel execution of tests

 docs/devel/testing.rst                   | 12 +++++++
 pythondeps.toml                          |  2 +-
 tests/Makefile.include                   |  6 +++-
 tests/avocado/boot_linux_console.py      |  1 -
 tests/avocado/boot_xen.py                | 13 ++++----
 tests/avocado/cdrom.py                   | 41 ++++++++++++++++++++++++
 tests/avocado/intel_iommu.py             |  1 -
 tests/avocado/kvm_xen_guest.py           |  2 +-
 tests/avocado/linux_initrd.py            |  1 -
 tests/avocado/machine_aarch64_sbsaref.py |  4 +--
 tests/avocado/machine_aarch64_virt.py    | 14 ++++----
 tests/avocado/machine_aspeed.py          |  2 --
 tests/avocado/machine_mips_malta.py      |  2 --
 tests/avocado/machine_rx_gdbsim.py       |  2 --
 tests/avocado/netdev-ethtool.py          |  2 +-
 tests/avocado/reverse_debugging.py       |  4 ---
 tests/avocado/smmu.py                    |  1 -
 tests/avocado/tuxrun_baselines.py        | 16 ++++-----
 18 files changed, 82 insertions(+), 44 deletions(-)
 create mode 100644 tests/avocado/cdrom.py

-- 
2.45.2

[PATCH v2 1/9] Bump avocado to 103.0

[Cleber Rosa]

This bumps Avocado to latest the LTS release.

An LTS release is one that can receive bugfixes and guarantees
stability for a much longer period and has incremental minor releases
made.

Even though the 103.0 LTS release is pretty a rewrite of Avocado when
compared to 88.1, the behavior of all existing tests under
tests/avocado has been extensively tested no regression in behavior
was found.

To keep behavior of jobs as close as possible with previous version,
this version bump keeps the execution serial (maximum of one task at a
time being run).

Reference: https://avocado-framework.readthedocs.io/en/103.0/releases/lts/103_0.html
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 pythondeps.toml        | 2 +-
 tests/Makefile.include | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/pythondeps.toml b/pythondeps.toml
index f6e590fdd8..175cf99241 100644
--- a/pythondeps.toml
+++ b/pythondeps.toml
@@ -30,5 +30,5 @@ sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
 # Note that qemu.git/python/ is always implicitly installed.
 # Prefer an LTS version when updating the accepted versions of
 # avocado-framework, for example right now the limit is 92.x.
-avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
+avocado-framework = { accepted = "(>=103.0, <104.0)", installed = "103.0", canary = "avocado" }
 pycdlib = { accepted = ">=1.11.0" }
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 6618bfed70..537804d101 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -141,7 +141,7 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
             $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
 			--filter-by-tags-include-empty-key) \
-            $(AVOCADO_CMDLINE_TAGS) \
+            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \
             $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
             "AVOCADO", "tests/avocado")
 
-- 
2.45.2

[PATCH v2 2/9] tests/avocado: apply proper skipUnless decorator

[Cleber Rosa]

Commit 9b45cc993 added many cases of skipUnless for the sake of
organizing flaky tests.  But, Python decorators *must* follow what
they decorate, so the newlines added should *not* exist there.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/boot_linux_console.py | 1 -
 tests/avocado/intel_iommu.py        | 1 -
 tests/avocado/linux_initrd.py       | 1 -
 tests/avocado/machine_aspeed.py     | 2 --
 tests/avocado/machine_mips_malta.py | 2 --
 tests/avocado/machine_rx_gdbsim.py  | 2 --
 tests/avocado/reverse_debugging.py  | 4 ----
 tests/avocado/smmu.py               | 1 -
 8 files changed, 14 deletions(-)

diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
index 2929aa042d..cffdd6b5a2 100644
--- a/tests/avocado/boot_linux_console.py
+++ b/tests/avocado/boot_linux_console.py
@@ -1522,7 +1522,6 @@ def test_ppc_mac99(self):
     # like issues with a buggy kernel. As a result we don't want it
     # gating releases on Gitlab.
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_sh4_r2d(self):
         """
         :avocado: tags=arch:sh4
diff --git a/tests/avocado/intel_iommu.py b/tests/avocado/intel_iommu.py
index 008f214397..992583fa7d 100644
--- a/tests/avocado/intel_iommu.py
+++ b/tests/avocado/intel_iommu.py
@@ -13,7 +13,6 @@
 from avocado_qemu.linuxtest import LinuxTest
 
 @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
 class IntelIOMMU(LinuxTest):
     """
     :avocado: tags=arch:x86_64
diff --git a/tests/avocado/linux_initrd.py b/tests/avocado/linux_initrd.py
index aad5b19bd9..7f47b98ae7 100644
--- a/tests/avocado/linux_initrd.py
+++ b/tests/avocado/linux_initrd.py
@@ -54,7 +54,6 @@ def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self):
             self.assertRegex(self.vm.get_log(), expected_msg)
 
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_with_2gib_file_should_work_with_linux_v4_16(self):
         """
         :avocado: tags=flaky
diff --git a/tests/avocado/machine_aspeed.py b/tests/avocado/machine_aspeed.py
index f8e263d37e..c0b01e8f1f 100644
--- a/tests/avocado/machine_aspeed.py
+++ b/tests/avocado/machine_aspeed.py
@@ -323,7 +323,6 @@ def do_test_aarch64_aspeed_sdk_start(self, image):
         self.wait_for_console_pattern('Starting kernel ...')
 
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_arm_ast2500_evb_sdk(self):
         """
         :avocado: tags=arch:arm
@@ -343,7 +342,6 @@ def test_arm_ast2500_evb_sdk(self):
         self.wait_for_console_pattern('nodistro.0 ast2500-default ttyS4')
 
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_arm_ast2600_evb_sdk(self):
         """
         :avocado: tags=arch:arm
diff --git a/tests/avocado/machine_mips_malta.py b/tests/avocado/machine_mips_malta.py
index 8cf84bd805..07a80633b5 100644
--- a/tests/avocado/machine_mips_malta.py
+++ b/tests/avocado/machine_mips_malta.py
@@ -102,7 +102,6 @@ def test_mips_malta_i6400_framebuffer_logo_1core(self):
         self.do_test_i6400_framebuffer_logo(1)
 
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_mips_malta_i6400_framebuffer_logo_7cores(self):
         """
         :avocado: tags=arch:mips64el
@@ -114,7 +113,6 @@ def test_mips_malta_i6400_framebuffer_logo_7cores(self):
         self.do_test_i6400_framebuffer_logo(7)
 
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_mips_malta_i6400_framebuffer_logo_8cores(self):
         """
         :avocado: tags=arch:mips64el
diff --git a/tests/avocado/machine_rx_gdbsim.py b/tests/avocado/machine_rx_gdbsim.py
index 412a7a5089..72f0296e21 100644
--- a/tests/avocado/machine_rx_gdbsim.py
+++ b/tests/avocado/machine_rx_gdbsim.py
@@ -23,7 +23,6 @@ class RxGdbSimMachine(QemuSystemTest):
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
 
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_uboot(self):
         """
         U-Boot and checks that the console is operational.
@@ -49,7 +48,6 @@ def test_uboot(self):
         #exec_command_and_wait_for_pattern(self, 'version', gcc_version)
 
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_linux_sash(self):
         """
         Boots a Linux kernel and checks that the console is operational.
diff --git a/tests/avocado/reverse_debugging.py b/tests/avocado/reverse_debugging.py
index 92855a02a5..f24287cd0a 100644
--- a/tests/avocado/reverse_debugging.py
+++ b/tests/avocado/reverse_debugging.py
@@ -207,7 +207,6 @@ def get_pc(self, g):
 
     # unidentified gitlab timeout problem
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_x86_64_pc(self):
         """
         :avocado: tags=arch:x86_64
@@ -225,7 +224,6 @@ class ReverseDebugging_AArch64(ReverseDebugging):
 
     # unidentified gitlab timeout problem
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_aarch64_virt(self):
         """
         :avocado: tags=arch:aarch64
@@ -250,7 +248,6 @@ class ReverseDebugging_ppc64(ReverseDebugging):
 
     # unidentified gitlab timeout problem
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_ppc64_pseries(self):
         """
         :avocado: tags=arch:ppc64
@@ -265,7 +262,6 @@ def test_ppc64_pseries(self):
 
     # See https://gitlab.com/qemu-project/qemu/-/issues/1992
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
     def test_ppc64_powernv(self):
         """
         :avocado: tags=arch:ppc64
diff --git a/tests/avocado/smmu.py b/tests/avocado/smmu.py
index aadda71e4b..83fd79e922 100644
--- a/tests/avocado/smmu.py
+++ b/tests/avocado/smmu.py
@@ -14,7 +14,6 @@
 from avocado_qemu.linuxtest import LinuxTest
 
 @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
-
 class SMMU(LinuxTest):
     """
     :avocado: tags=accel:kvm
-- 
2.45.2

Re: [PATCH v2 2/9] tests/avocado: apply proper skipUnless decorator

[Philippe Mathieu-Daudé]

On 6/8/24 19:31, Cleber Rosa wrote:
> Commit 9b45cc993 added many cases of skipUnless for the sake of
> organizing flaky tests.  But, Python decorators *must* follow what
> they decorate, so the newlines added should *not* exist there.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   tests/avocado/boot_linux_console.py | 1 -
>   tests/avocado/intel_iommu.py        | 1 -
>   tests/avocado/linux_initrd.py       | 1 -
>   tests/avocado/machine_aspeed.py     | 2 --
>   tests/avocado/machine_mips_malta.py | 2 --
>   tests/avocado/machine_rx_gdbsim.py  | 2 --
>   tests/avocado/reverse_debugging.py  | 4 ----
>   tests/avocado/smmu.py               | 1 -
>   8 files changed, 14 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH v2 2/9] tests/avocado: apply proper skipUnless decorator

[Thomas Huth]

On 06/08/2024 19.31, Cleber Rosa wrote:
> Commit 9b45cc993 added many cases of skipUnless for the sake of
> organizing flaky tests.  But, Python decorators *must* follow what

s/follow/directly precede/ ?

Apart from that:
Reviewed-by: Thomas Huth <thuth@redhat.com>

> they decorate, so the newlines added should *not* exist there.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   tests/avocado/boot_linux_console.py | 1 -
>   tests/avocado/intel_iommu.py        | 1 -
>   tests/avocado/linux_initrd.py       | 1 -
>   tests/avocado/machine_aspeed.py     | 2 --
>   tests/avocado/machine_mips_malta.py | 2 --
>   tests/avocado/machine_rx_gdbsim.py  | 2 --
>   tests/avocado/reverse_debugging.py  | 4 ----
>   tests/avocado/smmu.py               | 1 -
>   8 files changed, 14 deletions(-)

[PATCH v2 3/9] tests/avocado: add cdrom permission related tests

[Cleber Rosa]

The "media=cdrom" parameter is also used on some Avocado tests as a
way to protect files from being written.  The tests here bring a more
fundamental check that this behavior can be trusted.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/cdrom.py | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
 create mode 100644 tests/avocado/cdrom.py

diff --git a/tests/avocado/cdrom.py b/tests/avocado/cdrom.py
new file mode 100644
index 0000000000..c9aa5d69cb
--- /dev/null
+++ b/tests/avocado/cdrom.py
@@ -0,0 +1,41 @@
+# Simple functional tests for cdrom devices
+#
+# Copyright (c) 2023 Red Hat, Inc.
+#
+# Author:
+#  Cleber Rosa <crosa@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+
+from avocado.utils.iso9660 import iso9660
+from avocado_qemu import QemuSystemTest
+
+
+class Cdrom(QemuSystemTest):
+    """
+    :avocado: tags=block,cdrom,quick
+    :avocado: tags=machine:none
+    """
+    def setUp(self):
+        super().setUp()
+        self.iso_path = os.path.join(self.workdir, "cdrom.iso")
+        iso = iso9660(self.iso_path)
+        iso.create()
+        iso.close()
+
+    def test_plain_iso_rw(self):
+        self.vm.add_args('-drive', f'file={self.iso_path}')
+        self.vm.launch()
+        query_block_result = self.vm.qmp('query-block')['return']
+        self.assertEqual(len(query_block_result), 1)
+        self.assertFalse(query_block_result[0]["inserted"]["ro"])
+
+    def test_media_cdrom_ro(self):
+        self.vm.add_args('-drive', f'file={self.iso_path},media=cdrom')
+        self.vm.launch()
+        query_block_result = self.vm.qmp('query-block')['return']
+        self.assertEqual(len(query_block_result), 1)
+        self.assertTrue(query_block_result[0]["inserted"]["ro"])
-- 
2.45.2

Re: [PATCH v2 3/9] tests/avocado: add cdrom permission related tests

[Thomas Huth]

On 06/08/2024 19.31, Cleber Rosa wrote:
> The "media=cdrom" parameter is also used on some Avocado tests as a
> way to protect files from being written.  The tests here bring a more
> fundamental check that this behavior can be trusted.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   tests/avocado/cdrom.py | 41 +++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 41 insertions(+)
>   create mode 100644 tests/avocado/cdrom.py
> 
> diff --git a/tests/avocado/cdrom.py b/tests/avocado/cdrom.py
> new file mode 100644
> index 0000000000..c9aa5d69cb
> --- /dev/null
> +++ b/tests/avocado/cdrom.py
> @@ -0,0 +1,41 @@
> +# Simple functional tests for cdrom devices
> +#
> +# Copyright (c) 2023 Red Hat, Inc.
> +#
> +# Author:
> +#  Cleber Rosa <crosa@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +import os
> +
> +from avocado.utils.iso9660 import iso9660
> +from avocado_qemu import QemuSystemTest
> +
> +
> +class Cdrom(QemuSystemTest):
> +    """
> +    :avocado: tags=block,cdrom,quick
> +    :avocado: tags=machine:none
> +    """
> +    def setUp(self):
> +        super().setUp()
> +        self.iso_path = os.path.join(self.workdir, "cdrom.iso")
> +        iso = iso9660(self.iso_path)
> +        iso.create()
> +        iso.close()
> +
> +    def test_plain_iso_rw(self):
> +        self.vm.add_args('-drive', f'file={self.iso_path}')
> +        self.vm.launch()
> +        query_block_result = self.vm.qmp('query-block')['return']
> +        self.assertEqual(len(query_block_result), 1)
> +        self.assertFalse(query_block_result[0]["inserted"]["ro"])
> +
> +    def test_media_cdrom_ro(self):
> +        self.vm.add_args('-drive', f'file={self.iso_path},media=cdrom')
> +        self.vm.launch()
> +        query_block_result = self.vm.qmp('query-block')['return']
> +        self.assertEqual(len(query_block_result), 1)
> +        self.assertTrue(query_block_result[0]["inserted"]["ro"])

I think such a test would be a better fit for the qemu-iotests, e.g. 
tests/qemu-iotests/118 looks pretty similar already, maybe this could be 
added there instead?

  Thomas

[PATCH v2 4/9] tests/avocado: machine aarch64: standardize location and RO access

[Cleber Rosa]

The tests under machine_aarch64_virt.py and machine_aarch64_sbsaref.py
should not be writing to the ISO files.  By adding "media=cdrom" the
"ro" is autmatically set.

While at it, let's use a single code style and hash for the ISO url.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/machine_aarch64_sbsaref.py |  2 +-
 tests/avocado/machine_aarch64_virt.py    | 14 +++++++-------
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
index e920bbf08c..756f316ac9 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -129,7 +129,7 @@ def boot_alpine_linux(self, cpu):
             "-cpu",
             cpu,
             "-drive",
-            f"file={iso_path},format=raw",
+            f"file={iso_path},media=cdrom,format=raw",
         )
 
         self.vm.launch()
diff --git a/tests/avocado/machine_aarch64_virt.py b/tests/avocado/machine_aarch64_virt.py
index a90dc6ff4b..6831d2c0ed 100644
--- a/tests/avocado/machine_aarch64_virt.py
+++ b/tests/avocado/machine_aarch64_virt.py
@@ -37,13 +37,13 @@ def test_alpine_virt_tcg_gic_max(self):
         :avocado: tags=machine:virt
         :avocado: tags=accel:tcg
         """
-        iso_url = ('https://dl-cdn.alpinelinux.org/'
-                   'alpine/v3.17/releases/aarch64/'
-                   'alpine-standard-3.17.2-aarch64.iso')
+        iso_url = (
+            "https://dl-cdn.alpinelinux.org/"
+            "alpine/v3.17/releases/aarch64/alpine-standard-3.17.2-aarch64.iso"
+        )
 
-        # Alpine use sha256 so I recalculated this myself
-        iso_sha1 = '76284fcd7b41fe899b0c2375ceb8470803eea839'
-        iso_path = self.fetch_asset(iso_url, asset_hash=iso_sha1)
+        iso_hash = "5a36304ecf039292082d92b48152a9ec21009d3a62f459de623e19c4bd9dc027"
+        iso_path = self.fetch_asset(iso_url, algorithm="sha256", asset_hash=iso_hash)
 
         self.vm.set_console()
         kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
@@ -60,7 +60,7 @@ def test_alpine_virt_tcg_gic_max(self):
         self.vm.add_args("-smp", "2", "-m", "1024")
         self.vm.add_args('-bios', os.path.join(BUILD_DIR, 'pc-bios',
                                                'edk2-aarch64-code.fd'))
-        self.vm.add_args("-drive", f"file={iso_path},format=raw")
+        self.vm.add_args("-drive", f"file={iso_path},media=cdrom,format=raw")
         self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
         self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
 
-- 
2.45.2

[PATCH v2 5/9] tests/avocado: simplify parameters on fetch_asset with name only

[Cleber Rosa]

When an asset has a single location, it's possible to use that
URI as the name of the asset.

Reference: https://avocado-framework.readthedocs.io/en/103.0/api/utils/avocado.utils.html#avocado.utils.asset.Asset
Reference: https://avocado-framework.readthedocs.io/en/103.0/api/test/avocado.html#avocado.Test.fetch_asset

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/kvm_xen_guest.py  | 2 +-
 tests/avocado/netdev-ethtool.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/avocado/kvm_xen_guest.py b/tests/avocado/kvm_xen_guest.py
index f8cb458d5d..b6ee1ff752 100644
--- a/tests/avocado/kvm_xen_guest.py
+++ b/tests/avocado/kvm_xen_guest.py
@@ -40,7 +40,7 @@ def get_asset(self, name, sha1):
         url = base_url + name
         # use explicit name rather than failing to neatly parse the
         # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(name=url, asset_hash=sha1)
 
     def common_vm_setup(self):
         # We also catch lack of KVM_XEN support if we fail to launch
diff --git a/tests/avocado/netdev-ethtool.py b/tests/avocado/netdev-ethtool.py
index 5f33288f81..7345ded28c 100644
--- a/tests/avocado/netdev-ethtool.py
+++ b/tests/avocado/netdev-ethtool.py
@@ -27,7 +27,7 @@ def get_asset(self, name, sha1):
         url = base_url + name
         # use explicit name rather than failing to neatly parse the
         # URL into a unique one
-        return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+        return self.fetch_asset(name=url, asset_hash=sha1)
 
     def common_test_code(self, netdev, extra_args=None):
 
-- 
2.45.2

[PATCH v2 6/9] tests/avocado/boot_xen.py: fetch kernel during test setUp()

[Cleber Rosa]

The kernel is a common blob used in all tests.  By moving it to the
setUp() method, the "fetch asset" plugin will recognize the kernel and
attempt to fetch it and cache it before the tests are started.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/boot_xen.py | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/tests/avocado/boot_xen.py b/tests/avocado/boot_xen.py
index f29bc58b9e..490a127a3e 100644
--- a/tests/avocado/boot_xen.py
+++ b/tests/avocado/boot_xen.py
@@ -30,23 +30,22 @@ class BootXen(LinuxKernelTest):
     timeout = 90
     XEN_COMMON_COMMAND_LINE = 'dom0_mem=128M loglvl=all guest_loglvl=all'
 
-    def fetch_guest_kernel(self):
+    def setUp(self):
+        super(BootXen, self).setUp()
+
         # Using my own built kernel - which works
         kernel_url = ('https://fileserver.linaro.org/'
                       's/JSsewXGZ6mqxPr5/download?path=%2F&files='
                       'linux-5.9.9-arm64-ajb')
         kernel_sha1 = '4f92bc4b9f88d5ab792fa7a43a68555d344e1b83'
-        kernel_path = self.fetch_asset(kernel_url,
-                                       asset_hash=kernel_sha1)
-
-        return kernel_path
+        self.kernel_path = self.fetch_asset(kernel_url,
+                                            asset_hash=kernel_sha1)
 
     def launch_xen(self, xen_path):
         """
         Launch Xen with a dom0 guest kernel
         """
         self.log.info("launch with xen_path: %s", xen_path)
-        kernel_path = self.fetch_guest_kernel()
 
         self.vm.set_console()
 
@@ -56,7 +55,7 @@ def launch_xen(self, xen_path):
                          '-append', self.XEN_COMMON_COMMAND_LINE,
                          '-device',
                          'guest-loader,addr=0x47000000,kernel=%s,bootargs=console=hvc0'
-                         % (kernel_path))
+                         % (self.kernel_path))
 
         self.vm.launch()
 
-- 
2.45.2

[PATCH v2 7/9] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

[Cleber Rosa]

This makes use of the avocado.utils.archive support for zstd.

In order to not duplicate code, the skip condition uses a private
utility from the module which is going to become public in Avocado
versions 103.1 LTS (and also in versions >= 107.0).

Reference: https://github.com/avocado-framework/avocado/pull/5996
Reference: https://github.com/avocado-framework/avocado/pull/5953
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/tuxrun_baselines.py | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)

diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py
index 736e4aa289..bd02e88ed6 100644
--- a/tests/avocado/tuxrun_baselines.py
+++ b/tests/avocado/tuxrun_baselines.py
@@ -17,6 +17,7 @@
 from avocado_qemu import QemuSystemTest
 from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
 from avocado_qemu import wait_for_console_pattern
+from avocado.utils import archive
 from avocado.utils import process
 from avocado.utils.path import find_command
 
@@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None):
 
         return default
 
+    @skipUnless(archive._probe_zstd_cmd(),
+                'Could not find "zstd", or it is not able to properly '
+                'decompress decompress the rootfs')
     def setUp(self):
         super().setUp()
 
-        # We need zstd for all the tuxrun tests
-        # See https://github.com/avocado-framework/avocado/issues/5609
-        zstd = find_command('zstd', False)
-        if zstd is False:
-            self.cancel('Could not find "zstd", which is required to '
-                        'decompress rootfs')
-        self.zstd = zstd
-
         # Process the TuxRun specific tags, most machines work with
         # reasonable defaults but we sometimes need to tweak the
         # config. To avoid open coding everything we store all these
@@ -99,8 +95,8 @@ def fetch_tuxrun_assets(self, csums=None, dt=None):
                                          asset_hash = isum,
                                          algorithm = "sha256")
 
-        cmd = f"{self.zstd} -d {disk_image_zst} -o {self.workdir}/rootfs.ext4"
-        process.run(cmd)
+        archive.extract(disk_image_zst, os.path.join(self.workdir,
+                                                     "rootfs.ext4"))
 
         if dt:
             dsum = csums.get(dt, None)
-- 
2.45.2

Re: [PATCH v2 7/9] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

[Marcin Juszkiewicz]

On 6.08.2024 19:31, Cleber Rosa wrote:
> This makes use of the avocado.utils.archive support for zstd.
> 
> In order to not duplicate code, the skip condition uses a private
> utility from the module which is going to become public in Avocado
> versions 103.1 LTS (and also in versions >= 107.0).
> 
> Reference: https://github.com/avocado-framework/avocado/pull/5996
> Reference: https://github.com/avocado-framework/avocado/pull/5953
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   tests/avocado/tuxrun_baselines.py | 16 ++++++----------
>   1 file changed, 6 insertions(+), 10 deletions(-)
> 
> diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py
> index 736e4aa289..bd02e88ed6 100644
> --- a/tests/avocado/tuxrun_baselines.py
> +++ b/tests/avocado/tuxrun_baselines.py
> @@ -17,6 +17,7 @@
>   from avocado_qemu import QemuSystemTest
>   from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
>   from avocado_qemu import wait_for_console_pattern
> +from avocado.utils import archive
>   from avocado.utils import process
>   from avocado.utils.path import find_command
>   
> @@ -40,17 +41,12 @@ def get_tag(self, tagname, default=None):
>   
>           return default
>   
> +    @skipUnless(archive._probe_zstd_cmd(),
> +                'Could not find "zstd", or it is not able to properly '
> +                'decompress decompress the rootfs')

One "decompress" would be enough.

Re: [PATCH v2 7/9] tests/avocado/tuxrun_baselines.py: use Avocado's zstd support

[Cleber Rosa]

On Tue, Aug 6, 2024 at 2:20 PM Marcin Juszkiewicz
<marcin.juszkiewicz@linaro.org> wrote:
>
> On 6.08.2024 19:31, Cleber Rosa wrote:
> > +    @skipUnless(archive._probe_zstd_cmd(),
> > +                'Could not find "zstd", or it is not able to properly '
> > +                'decompress decompress the rootfs')
>
> One "decompress" would be enough.
>

True! Thanks for spotting that.

Regards,
- Cleber.

[PATCH v2 8/9] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image

[Cleber Rosa]

When the OpenBSD based tests are run in parallel, the previously
single instance of the image would become corrupt.  Let's give each
test its own snapshot.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/avocado/machine_aarch64_sbsaref.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
index 756f316ac9..f8bf40c192 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -190,7 +190,7 @@ def boot_openbsd73(self, cpu):
             "-cpu",
             cpu,
             "-drive",
-            f"file={img_path},format=raw",
+            f"file={img_path},format=raw,snapshot=on",
         )
 
         self.vm.launch()
-- 
2.45.2

Re: [PATCH v2 8/9] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image

[Philippe Mathieu-Daudé]

On 6/8/24 19:31, Cleber Rosa wrote:
> When the OpenBSD based tests are run in parallel, the previously
> single instance of the image would become corrupt.  Let's give each
> test its own snapshot.
>

Suggested-by: Alex Bennée <alex.bennee@linaro.org>
?

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   tests/avocado/machine_aarch64_sbsaref.py | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
> index 756f316ac9..f8bf40c192 100644
> --- a/tests/avocado/machine_aarch64_sbsaref.py
> +++ b/tests/avocado/machine_aarch64_sbsaref.py
> @@ -190,7 +190,7 @@ def boot_openbsd73(self, cpu):
>               "-cpu",
>               cpu,
>               "-drive",
> -            f"file={img_path},format=raw",
> +            f"file={img_path},format=raw,snapshot=on",
>           )
>   
>           self.vm.launch()

Re: [PATCH v2 8/9] tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image

[Cleber Rosa]

On Tue, Aug 6, 2024 at 3:30 PM Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>
> On 6/8/24 19:31, Cleber Rosa wrote:
> > When the OpenBSD based tests are run in parallel, the previously
> > single instance of the image would become corrupt.  Let's give each
> > test its own snapshot.
> >
>
> Suggested-by: Alex Bennée <alex.bennee@linaro.org>
> ?

Yes, sorry about missing that.

>
> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>

Thanks for the review.

Regards,
- Cleber

[PATCH v2 9/9] Avocado tests: allow for parallel execution of tests

[Cleber Rosa]

The updated Avocado version allows for the execution of tests in
parallel.

While on a CI environment it may not be a good idea to increase the
parallelization level in a single runner, developers may leverage that
on specific CI runners or on their development environments.

This also multiplies the timeout for each test accordingly.  The
reason is that more concurrency can lead to less resources, and less
resources can lead to some specific tests taking longer to complete
and then time out.  The timeout factor being used here is very
conservative (being equal to the amount of parallel tasks).  The worst
this possibly oversized timeout value can do is making users wait a
bit longer for the job to finish if a test hangs.

Overall, users can expect a much quicker turnaround on most systems
with a value such as 8 on a 12 core machine.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 docs/devel/testing.rst | 12 ++++++++++++
 tests/Makefile.include |  6 +++++-
 2 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index af73d3d64f..97ebc8211f 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -998,6 +998,18 @@ of Avocado or ``make check-avocado``, and can also be queried using:
 
   pyvenv/bin/avocado list tests/avocado
 
+To run tests in parallel, the ``AVOCADO_PARALLEL`` environment
+variable can be defined with a value different than ``1`` (its default
+value).  Example:
+
+ .. code::
+
+  make check-avocado AVOCADO_PARALLEL=4
+
+Please exercise care when using parallel execution with the QEMU
+Avocado tests as a higher system load can cause time sensitive tests
+to timeout and be interrupted.
+
 Manual Installation
 ~~~~~~~~~~~~~~~~~~~
 
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 537804d101..545b5155f9 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 ifndef AVOCADO_TESTS
 	AVOCADO_TESTS=tests/avocado
 endif
+ifndef AVOCADO_PARALLEL
+	AVOCADO_PARALLEL=1
+endif
 # Controls the output generated by Avocado when running tests.
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
@@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
             $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
 			--filter-by-tags-include-empty-key) \
-            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \
+            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \
+			-p timeout_factor=$(AVOCADO_PARALLEL) \
             $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
             "AVOCADO", "tests/avocado")
 
-- 
2.45.2

Re: [PATCH v2 9/9] Avocado tests: allow for parallel execution of tests

[Thomas Huth]

On 06/08/2024 19.31, Cleber Rosa wrote:
> The updated Avocado version allows for the execution of tests in
> parallel.
> 
> While on a CI environment it may not be a good idea to increase the
> parallelization level in a single runner, developers may leverage that
> on specific CI runners or on their development environments.
> 
> This also multiplies the timeout for each test accordingly.  The
> reason is that more concurrency can lead to less resources, and less
> resources can lead to some specific tests taking longer to complete
> and then time out.  The timeout factor being used here is very
> conservative (being equal to the amount of parallel tasks).  The worst
> this possibly oversized timeout value can do is making users wait a
> bit longer for the job to finish if a test hangs.
> 
> Overall, users can expect a much quicker turnaround on most systems
> with a value such as 8 on a 12 core machine.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
...
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 537804d101..545b5155f9 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>   ifndef AVOCADO_TESTS
>   	AVOCADO_TESTS=tests/avocado
>   endif
> +ifndef AVOCADO_PARALLEL
> +	AVOCADO_PARALLEL=1
> +endif
>   # Controls the output generated by Avocado when running tests.
>   # Any number of command separated loggers are accepted.  For more
>   # information please refer to "avocado --help".
> @@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>               --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>               $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>   			--filter-by-tags-include-empty-key) \
> -            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \
> +            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \
> +			-p timeout_factor=$(AVOCADO_PARALLEL) \
>               $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>               "AVOCADO", "tests/avocado")

I think it was nicer in the previous attempt to bump the avocado version:

https://gitlab.com/qemu-project/qemu/-/commit/ec5ffa0056389c3c10ea2de1e783

This re-used the "-j" option from "make", so you could do "make -j$(nproc) 
check-avocado" just like with the other "check" targets.

  Thomas

Re: [PATCH v2 9/9] Avocado tests: allow for parallel execution of tests

[Cleber Rosa]

On Mon, Aug 12, 2024 at 6:17 AM Thomas Huth <thuth@redhat.com> wrote:
> ...
> > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > index 537804d101..545b5155f9 100644
> > --- a/tests/Makefile.include
> > +++ b/tests/Makefile.include
> > @@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> >   ifndef AVOCADO_TESTS
> >       AVOCADO_TESTS=tests/avocado
> >   endif
> > +ifndef AVOCADO_PARALLEL
> > +     AVOCADO_PARALLEL=1
> > +endif
> >   # Controls the output generated by Avocado when running tests.
> >   # Any number of command separated loggers are accepted.  For more
> >   # information please refer to "avocado --help".
> > @@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> >               --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> >               $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> >                       --filter-by-tags-include-empty-key) \
> > -            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \
> > +            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \
> > +                     -p timeout_factor=$(AVOCADO_PARALLEL) \
> >               $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> >               "AVOCADO", "tests/avocado")
>
> I think it was nicer in the previous attempt to bump the avocado version:
>
> https://gitlab.com/qemu-project/qemu/-/commit/ec5ffa0056389c3c10ea2de1e783
>
> This re-used the "-j" option from "make", so you could do "make -j$(nproc)
> check-avocado" just like with the other "check" targets.
>

Hi Thomas,

I can see why it looks better, but in practice, I'm not getting the
best behavior with such a change.

First, the fact that it enables the parallelization by default, while
there still seems to be issues with test timeout issues, and even
existing races between tests (which this series tried to address as
much as possible) will not result in the best experience IMO.  On my
12 core machine, and also on GitLab CI, having 4 tests running in
parallel gets a nice speed up (as others have reported) while still
being very stable.

I'd say making the number of parallel tests equal to `nproc` is best
kept for a future round.

Let me know if this sounds reasonable to you.

Regards,
- Cleber.

>   Thomas
>

Re: [PATCH v2 9/9] Avocado tests: allow for parallel execution of tests

[Thomas Huth]

On 15/08/2024 16.08, Cleber Rosa wrote:
> On Mon, Aug 12, 2024 at 6:17 AM Thomas Huth <thuth@redhat.com> wrote:
>> ...
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index 537804d101..545b5155f9 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -94,6 +94,9 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>>    ifndef AVOCADO_TESTS
>>>        AVOCADO_TESTS=tests/avocado
>>>    endif
>>> +ifndef AVOCADO_PARALLEL
>>> +     AVOCADO_PARALLEL=1
>>> +endif
>>>    # Controls the output generated by Avocado when running tests.
>>>    # Any number of command separated loggers are accepted.  For more
>>>    # information please refer to "avocado --help".
>>> @@ -141,7 +144,8 @@ check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>>>                --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>>>                $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>>>                        --filter-by-tags-include-empty-key) \
>>> -            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=1 \
>>> +            $(AVOCADO_CMDLINE_TAGS) --max-parallel-tasks=$(AVOCADO_PARALLEL) \
>>> +                     -p timeout_factor=$(AVOCADO_PARALLEL) \
>>>                $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>>>                "AVOCADO", "tests/avocado")
>>
>> I think it was nicer in the previous attempt to bump the avocado version:
>>
>> https://gitlab.com/qemu-project/qemu/-/commit/ec5ffa0056389c3c10ea2de1e783
>>
>> This re-used the "-j" option from "make", so you could do "make -j$(nproc)
>> check-avocado" just like with the other "check" targets.
>>
> 
> Hi Thomas,
> 
> I can see why it looks better, but in practice, I'm not getting the
> best behavior with such a change.
> 
> First, the fact that it enables the parallelization by default, while
> there still seems to be issues with test timeout issues, and even
> existing races between tests (which this series tried to address as
> much as possible) will not result in the best experience IMO.  On my
> 12 core machine, and also on GitLab CI, having 4 tests running in
> parallel gets a nice speed up (as others have reported) while still
> being very stable.
> 
> I'd say making the number of parallel tests equal to `nproc` is best
> kept for a future round.
> 
> Let me know if this sounds reasonable to you.

  Hi Cleber,

that patch that I linked did not set the default number of parallel tests to 
$(nproc), it just used the value of the "-j" option of make. So if you just 
run "make check-avocado" there, you only get single threaded execution as 
before. You explicitely have to run "make -jX check-avocado" to get X 
parallel threads. IMHO using "-j" is more intuitive than using yet another 
environment variable.

  Thomas

Re: [PATCH v2 9/9] Avocado tests: allow for parallel execution of tests

[Richard Henderson]

On 8/16/24 02:02, Thomas Huth wrote:
> that patch that I linked did not set the default number of parallel tests to $(nproc), it 
> just used the value of the "-j" option of make. So if you just run "make check-avocado" 
> there, you only get single threaded execution as before. You explicitely have to run "make 
> -jX check-avocado" to get X parallel threads. IMHO using "-j" is more intuitive than using 
> yet another environment variable.

Seconded.


r~

Re: [PATCH v2 0/9] Bump Avocado to 103.0 LTS and update tests for compatibility and new features

[Marcin Juszkiewicz]

On 6.08.2024 19:31, Cleber Rosa wrote:
> This is along  overdue update of the Avocado version used in QEMU.
> It comes a time where the role of the runner and the libraries are
> being discussed and questioned.
> 
> These exact commits have been staging on my side for over 30 days now,
> and I was exceeding what I should in terms of testing before posting.
> I apologize for the miscalculation.
> 
> Nevertheless, as pointed out, on the ML, these changes are needed NOW.

Tested-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>

SBSA Reference Platform tests can be done in 1/3rd of time is a nice update.

Serial run:
real    6m20,324s
user    12m18,446s
sys     0m36,686s

Parallel (4) run:
real    2m22,658s
user    11m50,514s
sys     0m26,088s