[PATCH v4 0/6] testing/next: avocado, docs, gitlab

[PATCH v4 0/6] testing/next: avocado, docs, gitlab

[Alex Bennée]

This is the current state of my testing tree. It contains a few
patches that didn't make it into 8.0 and more testing fixes including
a bump to the avocado release including Thomas' --max-parallel-tasks
hack to avoid running the tests in parallel. So far it doesn't look
like its causing any problems although I am seeing some timeouts on
the BSDs.

The following patches need review:

tests/avocado: use the new snapshots for testing
tests/requirements.txt: bump up avocado-framework version to 101.0 (0 acks, 1 sobs, 1 tbs)

Alex Bennée (2):
  tests/avocado: use the new snapshots for testing
  qemu-options: finesse the recommendations around -blockdev

Kautuk Consul (1):
  tests/requirements.txt: bump up avocado-framework version to 101.0

Philippe Mathieu-Daudé (1):
  tests/avocado: Add set of boot tests on SBSA-ref

Thomas Huth (2):
  gitlab-ci: Avoid to re-run "configure" in the device-crash-test jobs
  scripts/device-crash-test: Add a parameter to run with TCG only

 MAINTAINERS                              |   1 +
 .gitlab-ci.d/buildtest.yml               |   6 +-
 qemu-options.hx                          |  24 +++-
 scripts/device-crash-test                |   4 +-
 tests/Makefile.include                   |  18 ++-
 tests/avocado/machine_aarch64_sbsaref.py | 158 +++++++++++++++++++++
 tests/avocado/tuxrun_baselines.py        | 170 +++++++++++++++++++----
 tests/requirements.txt                   |   2 +-
 8 files changed, 343 insertions(+), 40 deletions(-)
 create mode 100644 tests/avocado/machine_aarch64_sbsaref.py

-- 
2.39.2

[PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Alex Bennée]

From: Kautuk Consul <kconsul@linux.vnet.ibm.com>

Avocado version 101.0 has a fix to re-compute the checksum
of an asset file if the algorithm used in the *-CHECKSUM
file isn't the same as the one being passed to it by the
avocado user (i.e. the avocado_qemu python module).
In the earlier avocado versions this fix wasn't there due
to which if the checksum wouldn't match the earlier
checksum (calculated by a different algorithm), the avocado
code would start downloading a fresh image from the internet
URL thus making the test-cases take longer to execute.

Bump up the avocado-framework version to 101.0.

Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>

---
v2
  - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
---
 tests/Makefile.include | 18 +++++++++++-------
 tests/requirements.txt |  2 +-
 2 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 9422ddaece..a4de0ad5a2 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
 # download all vm images, according to defined targets
 get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
 
+JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
+
 check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
-	$(call quiet-command, \
-            $(TESTS_PYTHON) -m avocado \
-            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
-            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
-			--filter-by-tags-include-empty-key) \
-            $(AVOCADO_CMDLINE_TAGS) \
-            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
+	$(call quiet-command, 							\
+            $(TESTS_PYTHON) -m avocado 						\
+            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) 	\
+            $(if $(AVOCADO_TAGS),, 						\
+			--filter-by-tags-include-empty 				\
+			--filter-by-tags-include-empty-key) 			\
+		--max-parallel-tasks $(JOBS_OPTION:-j%=%) 			\
+            $(AVOCADO_CMDLINE_TAGS) 						\
+            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), 			\
             "AVOCADO", "tests/avocado")
 
 check-acceptance-deprecated-warning:
diff --git a/tests/requirements.txt b/tests/requirements.txt
index 0ba561b6bd..a6f73da681 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -2,5 +2,5 @@
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
 # Note that qemu.git/python/ is always implicitly installed.
-avocado-framework==88.1
+avocado-framework==101.0
 pycdlib==1.11.0
-- 
2.39.2

[PATCH v4 2/6] tests/avocado: use the new snapshots for testing

[Alex Bennée]

The tuxboot images now have a stable snapshot URL so we can enable the
checksums and remove the avocado warnings. We will have to update as
old snapshots retire but that won't be too frequent.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/avocado/tuxrun_baselines.py | 170 +++++++++++++++++++++++++-----
 1 file changed, 144 insertions(+), 26 deletions(-)

diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py
index d343376faa..e8749717e9 100644
--- a/tests/avocado/tuxrun_baselines.py
+++ b/tests/avocado/tuxrun_baselines.py
@@ -77,20 +77,33 @@ def wait_for_console_pattern(self, success_message, vm=None):
                                  failure_message='Kernel panic - not syncing',
                                  vm=vm)
 
-    def fetch_tuxrun_assets(self, dt=None):
+    def fetch_tuxrun_assets(self, csums=None, dt=None):
         """
         Fetch the TuxBoot assets. They are stored in a standard way so we
         use the per-test tags to fetch details.
         """
-        base_url = f"https://storage.tuxboot.com/{self.tuxboot}/"
-        kernel_image =  self.fetch_asset(base_url + self.image)
-        disk_image_zst = self.fetch_asset(base_url + "rootfs.ext4.zst")
+        base_url = f"https://storage.tuxboot.com/20230331/{self.tuxboot}/"
+
+        # empty hash if we weren't passed one
+        csums = {} if csums is None else csums
+        ksum = csums.get(self.image, None)
+        isum = csums.get("rootfs.ext4.zst", None)
+
+        kernel_image =  self.fetch_asset(base_url + self.image,
+                                         asset_hash = ksum,
+                                         algorithm = "sha256")
+        disk_image_zst = self.fetch_asset(base_url + "rootfs.ext4.zst",
+                                         asset_hash = isum,
+                                         algorithm = "sha256")
 
         cmd = f"{self.zstd} -d {disk_image_zst} -o {self.workdir}/rootfs.ext4"
         process.run(cmd)
 
         if dt:
-            dtb = self.fetch_asset(base_url + dt)
+            dsum = csums.get(dt, None)
+            dtb = self.fetch_asset(base_url + dt,
+                                   asset_hash = dsum,
+                                   algorithm = "sha256")
         else:
             dtb = None
 
@@ -149,7 +162,9 @@ def run_tuxtest_tests(self, haltmsg):
         else:
             self.vm.wait()
 
-    def common_tuxrun(self, dt=None,
+    def common_tuxrun(self,
+                      csums=None,
+                      dt=None,
                       drive="virtio-blk-device",
                       haltmsg="reboot: System halted",
                       console_index=0):
@@ -158,7 +173,7 @@ def common_tuxrun(self, dt=None,
         special with the command line we can process most things using
         the tag metadata.
         """
-        (kernel, disk, dtb) = self.fetch_tuxrun_assets(dt)
+        (kernel, disk, dtb) = self.fetch_tuxrun_assets(csums, dt)
 
         self.prepare_run(kernel, disk, drive, dtb, console_index)
         self.vm.launch()
@@ -182,7 +197,11 @@ def test_arm64(self):
         :avocado: tags=console:ttyAMA0
         :avocado: tags=shutdown:nowait
         """
-        self.common_tuxrun()
+        sums = {"Image" :
+                "ce95a7101a5fecebe0fe630deee6bd97b32ba41bc8754090e9ad8961ea8674c7",
+                "rootfs.ext4.zst" :
+                "bbd5ed4b9c7d3f4ca19ba71a323a843c6b585e880115df3b7765769dbd9dd061"}
+        self.common_tuxrun(csums=sums)
 
     def test_arm64be(self):
         """
@@ -194,7 +213,11 @@ def test_arm64be(self):
         :avocado: tags=console:ttyAMA0
         :avocado: tags=shutdown:nowait
         """
-        self.common_tuxrun()
+        sums = { "Image" :
+                 "e0df4425eb2cd9ea9a283e808037f805641c65d8fcecc8f6407d8f4f339561b4",
+                 "rootfs.ext4.zst" :
+                 "e6ffd8813c8a335bc15728f2835f90539c84be7f8f5f691a8b01451b47fb4bd7"}
+        self.common_tuxrun(csums=sums)
 
     def test_armv5(self):
         """
@@ -206,7 +229,15 @@ def test_armv5(self):
         :avocado: tags=console:ttyAMA0
         :avocado: tags=shutdown:nowait
         """
-        self.common_tuxrun(drive="virtio-blk-pci",
+        sums = { "rootfs.ext4.zst" :
+                 "17177afa74e7294da0642861f08c88ca3c836764299a54bf6d1ce276cb9712a5",
+                 "versatile-pb.dtb" :
+                 "0bc0c0b0858cefd3c32b385c0d66d97142ded29472a496f4f490e42fc7615b25",
+                 "zImage" :
+                 "c95af2f27647c12265d75e9df44c22ff5228c59855f54aaa70f41ec2842e3a4d" }
+
+        self.common_tuxrun(csums=sums,
+                           drive="virtio-blk-pci",
                            dt="versatile-pb.dtb")
 
     def test_armv7(self):
@@ -219,7 +250,12 @@ def test_armv7(self):
         :avocado: tags=console:ttyAMA0
         :avocado: tags=shutdown:nowait
         """
-        self.common_tuxrun()
+        sums = { "rootfs.ext4.zst" :
+                 "ab1fbbeaddda1ffdd45c9405a28cd5370c20f23a7cbc809cc90dc9f243a8eb5a",
+                 "zImage" :
+                 "4c7a22e9f15875bec06bd2a29d822496571eb297d4f22694099ffcdb19077572" }
+
+        self.common_tuxrun(csums=sums)
 
     def test_armv7be(self):
         """
@@ -232,7 +268,12 @@ def test_armv7be(self):
         :avocado: tags=console:ttyAMA0
         :avocado: tags=shutdown:nowait
         """
-        self.common_tuxrun()
+        sums = {"rootfs.ext4.zst" :
+                "42ed46dd2d59986206c5b1f6cf35eab58fe3fd20c96b41aaa16b32f3f90a9835",
+                "zImage" :
+                "7facc62082b57af12015b08f7fdbaf2f123ba07a478367853ae12b219afc9f2f" }
+
+        self.common_tuxrun(csums=sums)
 
     def test_i386(self):
         """
@@ -243,7 +284,12 @@ def test_i386(self):
         :avocado: tags=image:bzImage
         :avocado: tags=shutdown:nowait
         """
-        self.common_tuxrun(drive="virtio-blk-pci")
+        sums = {"bzImage" :
+                "a3e5b32a354729e65910f5a1ffcda7c14a6c12a55e8213fb86e277f1b76ed956",
+                "rootfs.ext4.zst" :
+                "f15e66b2bf673a210ec2a4b2e744a80530b36289e04f5388aab812b97f69754a" }
+
+        self.common_tuxrun(csums=sums, drive="virtio-blk-pci")
 
     def test_mips32(self):
         """
@@ -256,7 +302,12 @@ def test_mips32(self):
         :avocado: tags=root:sda
         :avocado: tags=shutdown:nowait
         """
-        self.common_tuxrun(drive="driver=ide-hd,bus=ide.0,unit=0")
+        sums = { "rootfs.ext4.zst" :
+                 "fc3da0b4c2f38d74c6d705123bb0f633c76ed953128f9d0859378c328a6d11a0",
+                 "vmlinux" :
+                 "bfd2172f8b17fb32970ca0c8c58f59c5a4ca38aa5855d920be3a69b5d16e52f0" }
+
+        self.common_tuxrun(csums=sums, drive="driver=ide-hd,bus=ide.0,unit=0")
 
     def test_mips32el(self):
         """
@@ -268,7 +319,12 @@ def test_mips32el(self):
         :avocado: tags=root:sda
         :avocado: tags=shutdown:nowait
         """
-        self.common_tuxrun(drive="driver=ide-hd,bus=ide.0,unit=0")
+        sums = { "rootfs.ext4.zst" :
+                 "e799768e289fd69209c21f4dacffa11baea7543d5db101e8ce27e3bc2c41d90e",
+                 "vmlinux" :
+                 "8573867c68a8443db8de6d08bb33fb291c189ca2ca671471d3973a3e712096a3" }
+
+        self.common_tuxrun(csums=sums, drive="driver=ide-hd,bus=ide.0,unit=0")
 
     def test_mips64(self):
         """
@@ -280,7 +336,12 @@ def test_mips64(self):
         :avocado: tags=root:sda
         :avocado: tags=shutdown:nowait
         """
-        self.common_tuxrun(drive="driver=ide-hd,bus=ide.0,unit=0")
+        sums = { "rootfs.ext4.zst" :
+                 "69d91eeb04df3d8d172922c6993bb37d4deeb6496def75d8580f6f9de3e431da",
+                 "vmlinux" :
+                 "09010e51e4b8bcbbd2494786ffb48eca78f228e96e5c5438344b0eac4029dc61" }
+
+        self.common_tuxrun(csums=sums, drive="driver=ide-hd,bus=ide.0,unit=0")
 
     def test_mips64el(self):
         """
@@ -291,7 +352,12 @@ def test_mips64el(self):
         :avocado: tags=root:sda
         :avocado: tags=shutdown:nowait
         """
-        self.common_tuxrun(drive="driver=ide-hd,bus=ide.0,unit=0")
+        sums = { "rootfs.ext4.zst" :
+                 "fba585368f5915b1498ed081863474b2d7ec4e97cdd46d21bdcb2f9698f83de4",
+                 "vmlinux" :
+                 "d4e08965e2155c4cccce7c5f34d18fe34c636cda2f2c9844387d614950155266" }
+
+        self.common_tuxrun(csums=sums, drive="driver=ide-hd,bus=ide.0,unit=0")
 
     def test_ppc32(self):
         """
@@ -302,7 +368,12 @@ def test_ppc32(self):
         :avocado: tags=image:uImage
         :avocado: tags=shutdown:nowait
         """
-        self.common_tuxrun(drive="virtio-blk-pci")
+        sums = { "rootfs.ext4.zst" :
+                 "8885b9d999cc24d679542a02e9b6aaf48f718f2050ece6b8347074b6ee41dd09",
+                 "uImage" :
+                 "1a68f74b860fda022fb12e03c5efece8c2b8b590d96cca37a8481a3ae0b3f81f" }
+
+        self.common_tuxrun(csums=sums, drive="virtio-blk-pci")
 
     def test_ppc64(self):
         """
@@ -316,7 +387,12 @@ def test_ppc64(self):
         :avocado: tags=extradev:driver=spapr-vscsi
         :avocado: tags=root:sda
         """
-        self.common_tuxrun(drive="scsi-hd")
+        sums = { "rootfs.ext4.zst" :
+                 "1d953e81a4379e537fc8e41e05a0a59d9b453eef97aa03d47866c6c45b00bdff",
+                 "vmlinux" :
+                 "f22a9b9e924174a4c199f4c7e5d91a2339fcfe51c6eafd0907dc3e09b64ab728" }
+
+        self.common_tuxrun(csums=sums, drive="scsi-hd")
 
     def test_ppc64le(self):
         """
@@ -329,7 +405,12 @@ def test_ppc64le(self):
         :avocado: tags=extradev:driver=spapr-vscsi
         :avocado: tags=root:sda
         """
-        self.common_tuxrun(drive="scsi-hd")
+        sums = { "rootfs.ext4.zst" :
+                 "b442678c93fb8abe1f7d3bfa20556488de6b475c22c8fed363f42cf81a0a3906",
+                 "vmlinux" :
+                 "979eb61b445a010fb13e2b927126991f8ceef9c590fa2be0996c00e293e80cf2" }
+
+        self.common_tuxrun(csums=sums, drive="scsi-hd")
 
     def test_riscv32(self):
         """
@@ -337,7 +418,14 @@ def test_riscv32(self):
         :avocado: tags=machine:virt
         :avocado: tags=tuxboot:riscv32
         """
-        self.common_tuxrun()
+        sums = { "Image" :
+                 "89599407d7334de629a40e7ad6503c73670359eb5f5ae9d686353a3d6deccbd5",
+                 "fw_jump.elf" :
+                 "f2ef28a0b77826f79d085d3e4aa686f1159b315eff9099a37046b18936676985",
+                 "rootfs.ext4.zst" :
+                 "7168d296d0283238ea73cd5a775b3dd608e55e04c7b92b76ecce31bb13108cba" }
+
+        self.common_tuxrun(csums=sums)
 
     def test_riscv64(self):
         """
@@ -345,7 +433,14 @@ def test_riscv64(self):
         :avocado: tags=machine:virt
         :avocado: tags=tuxboot:riscv64
         """
-        self.common_tuxrun()
+        sums = { "Image" :
+                 "cd634badc65e52fb63465ec99e309c0de0369f0841b7d9486f9729e119bac25e",
+                 "fw_jump.elf" :
+                 "6e3373abcab4305fe151b564a4c71110d833c21f2c0a1753b7935459e36aedcf",
+                 "rootfs.ext4.zst" :
+                 "b18e3a3bdf27be03da0b285e84cb71bf09eca071c3a087b42884b6982ed679eb" }
+
+        self.common_tuxrun(csums=sums)
 
     def test_s390(self):
         """
@@ -355,7 +450,13 @@ def test_s390(self):
         :avocado: tags=image:bzImage
         :avocado: tags=shutdown:nowait
         """
-        self.common_tuxrun(drive="virtio-blk-ccw",
+        sums = { "bzImage" :
+                 "0414e98dd1c3dafff8496c9cd9c28a5f8d04553bb5ba37e906a812b48d442ef0",
+                 "rootfs.ext4.zst" :
+                 "88c37c32276677f873a25ab9ec6247895b8e3e6f8259134de2a616080b8ab3fc" }
+
+        self.common_tuxrun(csums=sums,
+                           drive="virtio-blk-ccw",
                            haltmsg="Requesting system halt")
 
     # Note: some segfaults caused by unaligned userspace access
@@ -370,9 +471,14 @@ def test_sh4(self):
         :avocado: tags=root:sda
         :avocado: tags=console:ttySC1
         """
+        sums = { "rootfs.ext4.zst" :
+                 "3592a7a3d5a641e8b9821449e77bc43c9904a56c30d45da0694349cfd86743fd",
+                 "zImage" :
+                 "29d9b2aba604a0f53a5dc3b5d0f2b8e35d497de1129f8ee5139eb6fdf0db692f" }
+
         # The test is currently too unstable to do much in userspace
         # so we skip common_tuxrun and do a minimal boot and shutdown.
-        (kernel, disk, dtb) = self.fetch_tuxrun_assets()
+        (kernel, disk, dtb) = self.fetch_tuxrun_assets(csums=sums)
 
         # the console comes on the second serial port
         self.prepare_run(kernel, disk,
@@ -395,7 +501,13 @@ def test_sparc64(self):
         :avocado: tags=root:sda
         :avocado: tags=shutdown:nowait
         """
-        self.common_tuxrun(drive="driver=ide-hd,bus=ide.0,unit=0")
+
+        sums = { "rootfs.ext4.zst" :
+                 "ad2f1dc436ab51583543d25d2c210cab478645d47078d30d129a66ab0e281d76",
+                 "vmlinux" :
+                 "e34313e4325ff21deaa3d38a502aa09a373ef62b9bd4d7f8f29388b688225c55" }
+
+        self.common_tuxrun(csums=sums, drive="driver=ide-hd,bus=ide.0,unit=0")
 
     def test_x86_64(self):
         """
@@ -407,4 +519,10 @@ def test_x86_64(self):
         :avocado: tags=root:sda
         :avocado: tags=shutdown:nowait
         """
-        self.common_tuxrun(drive="driver=ide-hd,bus=ide.0,unit=0")
+        sums = { "bzImage" :
+                 "2bc7480a669ee9b6b82500a236aba0c54233debe98cb968268fa230f52f03461",
+                 "rootfs.ext4.zst" :
+                 "b72ac729769b8f51c6dffb221113c9a063c774dbe1d66af30eb593c4e9999b4b" }
+
+        self.common_tuxrun(csums=sums,
+                           drive="driver=ide-hd,bus=ide.0,unit=0")
-- 
2.39.2

[PATCH v4 3/6] tests/avocado: Add set of boot tests on SBSA-ref

[Alex Bennée]

From: Philippe Mathieu-Daudé <philmd@linaro.org>

This change adds set of boot tests on SBSA-ref machine:

1. boot firmware up to the EDK2 banner
2. boot Alpine Linux

Prebuilt flash volumes are included, built using upstream documentation.

To unify tests for AArch64/virt and AArch64/sbsa-ref we boot
the same Alpine Linux image on both.

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230323082813.971535-1-marcin.juszkiewicz@linaro.org>
Reviewed-by: Leif Lindholm <quic_llindhol@quicinc.com>
Message-Id: <20230328171426.14258-1-philmd@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 MAINTAINERS                              |   1 +
 tests/avocado/machine_aarch64_sbsaref.py | 158 +++++++++++++++++++++++
 2 files changed, 159 insertions(+)
 create mode 100644 tests/avocado/machine_aarch64_sbsaref.py

diff --git a/MAINTAINERS b/MAINTAINERS
index 2c2068ea5c..b31e00f318 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -942,6 +942,7 @@ L: qemu-arm@nongnu.org
 S: Maintained
 F: hw/arm/sbsa-ref.c
 F: docs/system/arm/sbsa.rst
+F: tests/avocado/machine_aarch64_sbsaref.py
 
 Sharp SL-5500 (Collie) PDA
 M: Peter Maydell <peter.maydell@linaro.org>
diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
new file mode 100644
index 0000000000..0a79fa7ab6
--- /dev/null
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -0,0 +1,158 @@
+# Functional test that boots a Linux kernel and checks the console
+#
+# SPDX-FileCopyrightText: 2023 Linaro Ltd.
+# SPDX-FileContributor: Philippe Mathieu-Daudé <philmd@linaro.org>
+# SPDX-FileContributor: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+import os
+
+from avocado import skip
+from avocado import skipUnless
+from avocado.utils import archive
+
+from avocado_qemu import QemuSystemTest
+from avocado_qemu import wait_for_console_pattern
+from avocado_qemu import interrupt_interactive_console_until_pattern
+
+
+class Aarch64SbsarefMachine(QemuSystemTest):
+    """
+    :avocado: tags=arch:aarch64
+    :avocado: tags=machine:sbsa-ref
+    """
+
+    timeout = 180
+
+    def fetch_firmware(self):
+        """
+        Flash volumes generated using:
+
+        - Fedora GNU Toolchain version 12.2.1 20220819 (Red Hat Cross 12.2.1-2)
+
+        - Trusted Firmware-A
+          https://github.com/ARM-software/arm-trusted-firmware/tree/5fdb2e54
+
+        - Tianocore EDK II
+          https://github.com/tianocore/edk2/tree/494127613b
+          https://github.com/tianocore/edk2-non-osi/tree/41876073
+          https://github.com/tianocore/edk2-platforms/tree/8efa4f42
+        """
+
+        # Secure BootRom (TF-A code)
+        fs0_xz_url = (
+            "https://fileserver.linaro.org/s/ATnSmq6k8SoXgbH/"
+            "download/SBSA_FLASH0.fd.xz"
+        )
+        fs0_xz_hash = "a210a09692bcbe0a3743ffd0df44e80e0c7ad8ab"
+        tar_xz_path = self.fetch_asset(fs0_xz_url, asset_hash=fs0_xz_hash)
+        archive.extract(tar_xz_path, self.workdir)
+        fs0_path = os.path.join(self.workdir, "SBSA_FLASH0.fd")
+
+        # Non-secure rom (UEFI and EFI variables)
+        fs1_xz_url = (
+            "https://fileserver.linaro.org/s/t8foNnMPz74DZZy/"
+            "download/SBSA_FLASH1.fd.xz"
+        )
+        fs1_xz_hash = "13a9a262953787c7fc5a9155dfaa26e703631e02"
+        tar_xz_path = self.fetch_asset(fs1_xz_url, asset_hash=fs1_xz_hash)
+        archive.extract(tar_xz_path, self.workdir)
+        fs1_path = os.path.join(self.workdir, "SBSA_FLASH1.fd")
+
+        for path in [fs0_path, fs1_path]:
+            with open(path, "ab+") as fd:
+                fd.truncate(256 << 20)  # Expand volumes to 256MiB
+
+        self.vm.set_console()
+        self.vm.add_args(
+            "-drive",
+            f"if=pflash,file={fs0_path},format=raw",
+            "-drive",
+            f"if=pflash,file={fs1_path},format=raw",
+            "-smp",
+            "1",
+            "-machine",
+            "sbsa-ref",
+        )
+
+    def test_sbsaref_edk2_firmware(self):
+        """
+        :avocado: tags=cpu:cortex-a57
+        """
+
+        self.fetch_firmware()
+        self.vm.launch()
+
+        # TF-A boot sequence:
+        #
+        # https://github.com/ARM-software/arm-trusted-firmware/blob/v2.8.0/\
+        #     docs/design/trusted-board-boot.rst#trusted-board-boot-sequence
+        # https://trustedfirmware-a.readthedocs.io/en/v2.8/\
+        #     design/firmware-design.html#cold-boot
+
+        # AP Trusted ROM
+        wait_for_console_pattern(self, "Booting Trusted Firmware")
+        wait_for_console_pattern(self, "BL1: v2.8(release):v2.8")
+        wait_for_console_pattern(self, "BL1: Booting BL2")
+
+        # Trusted Boot Firmware
+        wait_for_console_pattern(self, "BL2: v2.8(release)")
+        wait_for_console_pattern(self, "Booting BL31")
+
+        # EL3 Runtime Software
+        wait_for_console_pattern(self, "BL31: v2.8(release)")
+
+        # Non-trusted Firmware
+        wait_for_console_pattern(self, "UEFI firmware (version 1.0")
+        interrupt_interactive_console_until_pattern(self, "QEMU SBSA-REF Machine")
+
+    # This tests the whole boot chain from EFI to Userspace
+    # We only boot a whole OS for the current top level CPU and GIC
+    # Other test profiles should use more minimal boots
+    def boot_alpine_linux(self, cpu):
+        self.fetch_firmware()
+
+        iso_url = (
+            "https://dl-cdn.alpinelinux.org/"
+            "alpine/v3.17/releases/aarch64/alpine-standard-3.17.2-aarch64.iso"
+        )
+
+        iso_hash = "5a36304ecf039292082d92b48152a9ec21009d3a62f459de623e19c4bd9dc027"
+        iso_path = self.fetch_asset(iso_url, algorithm="sha256", asset_hash=iso_hash)
+
+        self.vm.set_console()
+        self.vm.add_args(
+            "-cpu",
+            cpu,
+            "-drive",
+            f"file={iso_path},format=raw",
+            "-device",
+            "virtio-rng-pci,rng=rng0",
+            "-object",
+            "rng-random,id=rng0,filename=/dev/urandom",
+        )
+
+        self.vm.launch()
+        wait_for_console_pattern(self, "Welcome to Alpine Linux 3.17")
+
+    @skipUnless(os.getenv("AVOCADO_TIMEOUT_EXPECTED"), "Test might timeout")
+    def test_sbsaref_alpine_linux_cortex_a57(self):
+        """
+        :avocado: tags=cpu:cortex-a57
+        """
+        self.boot_alpine_linux("cortex-a57")
+
+    @skipUnless(os.getenv("AVOCADO_TIMEOUT_EXPECTED"), "Test might timeout")
+    def test_sbsaref_alpine_linux_neoverse_n1(self):
+        """
+        :avocado: tags=cpu:max
+        """
+        self.boot_alpine_linux("neoverse-n1")
+
+    @skip("requires TF-A update to handle FEAT_FGT")
+    def test_sbsaref_alpine_linux_max(self):
+        """
+        :avocado: tags=cpu:max
+        """
+        self.boot_alpine_linux("max,pauth-impdef=on")
-- 
2.39.2

[PATCH v4 4/6] gitlab-ci: Avoid to re-run "configure" in the device-crash-test jobs

[Alex Bennée]

From: Thomas Huth <thuth@redhat.com>

After "make check-venv" had been added to these jobs, they started
to re-run "configure" each time since our logic in the makefile
thinks that some files are out of date here. Avoid it with the same
trick that we are using in buildtest-template.yml already by disabling
the up-to-date check via NINJA=":".

Fixes: 1d8cf47e5b ("tests: run 'device-crash-test' from tests/venv")
Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230414145845.456145-2-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .gitlab-ci.d/buildtest.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index ba6f551752..333eea9dd3 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -102,7 +102,7 @@ crash-test-debian:
     IMAGE: debian-amd64
   script:
     - cd build
-    - make check-venv
+    - make NINJA=":" check-venv
     - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-i386
 
 build-system-fedora:
@@ -145,7 +145,7 @@ crash-test-fedora:
     IMAGE: fedora
   script:
     - cd build
-    - make check-venv
+    - make NINJA=":" check-venv
     - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc
     - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32
 
-- 
2.39.2

[PATCH v4 5/6] scripts/device-crash-test: Add a parameter to run with TCG only

[Alex Bennée]

From: Thomas Huth <thuth@redhat.com>

We're currently facing the problem that the device-crash-test script
runs twice as long in the CI when a runner supports KVM - which sometimes
results in a timeout of the CI job. To get a more deterministic runtime
here, add an option to the script that allows to run it with TCG only.

Reported-by: Eldon Stegall <eldon-qemu@eldondev.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230414145845.456145-3-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .gitlab-ci.d/buildtest.yml | 2 +-
 scripts/device-crash-test  | 4 +++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 333eea9dd3..bb3650a51c 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -103,7 +103,7 @@ crash-test-debian:
   script:
     - cd build
     - make NINJA=":" check-venv
-    - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-i386
+    - tests/venv/bin/python3 scripts/device-crash-test -q --tcg-only ./qemu-system-i386
 
 build-system-fedora:
   extends:
diff --git a/scripts/device-crash-test b/scripts/device-crash-test
index 73bcb98693..b74d887331 100755
--- a/scripts/device-crash-test
+++ b/scripts/device-crash-test
@@ -397,7 +397,7 @@ def binariesToTest(args, testcase):
 
 
 def accelsToTest(args, testcase):
-    if getBinaryInfo(args, testcase['binary']).kvm_available:
+    if getBinaryInfo(args, testcase['binary']).kvm_available and not args.tcg_only:
         yield 'kvm'
     yield 'tcg'
 
@@ -510,6 +510,8 @@ def main():
                         help="Full mode: test cases that are expected to fail")
     parser.add_argument('--strict', action='store_true', dest='strict',
                         help="Treat all warnings as fatal")
+    parser.add_argument('--tcg-only', action='store_true', dest='tcg_only',
+                        help="Only test with TCG accelerator")
     parser.add_argument('qemu', nargs='*', metavar='QEMU',
                         help='QEMU binary to run')
     args = parser.parse_args()
-- 
2.39.2

[PATCH v4 6/6] qemu-options: finesse the recommendations around -blockdev

[Alex Bennée]

We are a bit premature in recommending -blockdev/-device as the best
way to configure block devices. It seems there are times the more
human friendly -drive still makes sense especially when -snapshot is
involved.

Improve the language to hopefully make things clearer.

Suggested-by: Michael Tokarev <mjt@tls.msk.ru>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Cc: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20230330101141.30199-5-alex.bennee@linaro.org>

---
v3
  - more re-wording to try and approach consensus
  - add explicit warning to -snapshot option
---
 qemu-options.hx | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/qemu-options.hx b/qemu-options.hx
index 59bdf67a2c..4b8855a4f7 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -1143,10 +1143,22 @@ have gone through several iterations as the feature set and complexity
 of the block layer have grown. Many online guides to QEMU often
 reference older and deprecated options, which can lead to confusion.
 
-The recommended modern way to describe disks is to use a combination of
+The most explicit way to describe disks is to use a combination of
 ``-device`` to specify the hardware device and ``-blockdev`` to
 describe the backend. The device defines what the guest sees and the
-backend describes how QEMU handles the data.
+backend describes how QEMU handles the data. It is the only guaranteed
+stable interface for describing block devices and as such is
+recommended for management tools and scripting.
+
+The ``-drive`` option combines the device and backend into a single
+command line option which is a more human friendly. There is however no
+interface stability guarantee although some older board models still
+need updating to work with the modern blockdev forms.
+
+Older options like ``-hda`` are essentially macros which expand into
+``-drive`` options for various drive interfaces. The original forms
+bake in a lot of assumptions from the days when QEMU was emulating a
+legacy PC, they are not recommended for modern configurations.
 
 ERST
 
@@ -1639,6 +1651,14 @@ SRST
     the raw disk image you use is not written back. You can however
     force the write back by pressing C-a s (see the :ref:`disk images`
     chapter in the System Emulation Users Guide).
+
+    .. warning::
+       snapshot is incompatible with ``-blockdev`` (instead use qemu-img
+       to manually create snapshot images to attach to your blockdev).
+       If you have mixed ``-blockdev`` and ``-drive`` declarations you
+       can use the 'snapshot' property on your drive declarations
+       instead of this global option.
+
 ERST
 
 DEF("fsdev", HAS_ARG, QEMU_OPTION_fsdev,
-- 
2.39.2

Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[John Snow]

On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>
> Avocado version 101.0 has a fix to re-compute the checksum
> of an asset file if the algorithm used in the *-CHECKSUM
> file isn't the same as the one being passed to it by the
> avocado user (i.e. the avocado_qemu python module).
> In the earlier avocado versions this fix wasn't there due
> to which if the checksum wouldn't match the earlier
> checksum (calculated by a different algorithm), the avocado
> code would start downloading a fresh image from the internet
> URL thus making the test-cases take longer to execute.
>
> Bump up the avocado-framework version to 101.0.
>
> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>
> ---
> v2
>   - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> ---
>  tests/Makefile.include | 18 +++++++++++-------
>  tests/requirements.txt |  2 +-
>  2 files changed, 12 insertions(+), 8 deletions(-)
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 9422ddaece..a4de0ad5a2 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>  # download all vm images, according to defined targets
>  get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>
> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
> +
>  check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> -       $(call quiet-command, \
> -            $(TESTS_PYTHON) -m avocado \
> -            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> -                       --filter-by-tags-include-empty-key) \
> -            $(AVOCADO_CMDLINE_TAGS) \
> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> +       $(call quiet-command,                                                   \
> +            $(TESTS_PYTHON) -m avocado                                                 \
> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR)  \
> +            $(if $(AVOCADO_TAGS),,                                             \
> +                       --filter-by-tags-include-empty                          \
> +                       --filter-by-tags-include-empty-key)                     \
> +               --max-parallel-tasks $(JOBS_OPTION:-j%=%)                       \
> +            $(AVOCADO_CMDLINE_TAGS)                                            \
> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),                   \
>              "AVOCADO", "tests/avocado")
>
>  check-acceptance-deprecated-warning:
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index 0ba561b6bd..a6f73da681 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -2,5 +2,5 @@
>  # in the tests/venv Python virtual environment. For more info,
>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>  # Note that qemu.git/python/ is always implicitly installed.
> -avocado-framework==88.1
> +avocado-framework==101.0
>  pycdlib==1.11.0
> --
> 2.39.2
>

I thought there were test failures that prohibited us from bumping the
Avocado-Framework version. Did those get rectified recently?

Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Alex Bennée]

John Snow <jsnow@redhat.com> writes:

> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>
>> Avocado version 101.0 has a fix to re-compute the checksum
>> of an asset file if the algorithm used in the *-CHECKSUM
>> file isn't the same as the one being passed to it by the
>> avocado user (i.e. the avocado_qemu python module).
>> In the earlier avocado versions this fix wasn't there due
>> to which if the checksum wouldn't match the earlier
>> checksum (calculated by a different algorithm), the avocado
>> code would start downloading a fresh image from the internet
>> URL thus making the test-cases take longer to execute.
>>
>> Bump up the avocado-framework version to 101.0.
>>
>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>
>> ---
>> v2
>>   - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>> ---
>>  tests/Makefile.include | 18 +++++++++++-------
>>  tests/requirements.txt |  2 +-
>>  2 files changed, 12 insertions(+), 8 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 9422ddaece..a4de0ad5a2 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>  # download all vm images, according to defined targets
>>  get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>>
>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
>> +
>>  check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>> -       $(call quiet-command, \
>> -            $(TESTS_PYTHON) -m avocado \
>> -            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>> -                       --filter-by-tags-include-empty-key) \
>> -            $(AVOCADO_CMDLINE_TAGS) \
>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>> +       $(call quiet-command,                                                   \
>> +            $(TESTS_PYTHON) -m avocado                                                 \
>> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR)  \
>> +            $(if $(AVOCADO_TAGS),,                                             \
>> +                       --filter-by-tags-include-empty                          \
>> +                       --filter-by-tags-include-empty-key)                     \
>> +               --max-parallel-tasks $(JOBS_OPTION:-j%=%)                       \
>> +            $(AVOCADO_CMDLINE_TAGS)                                            \
>> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),                   \
>>              "AVOCADO", "tests/avocado")
>>
>>  check-acceptance-deprecated-warning:
>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>> index 0ba561b6bd..a6f73da681 100644
>> --- a/tests/requirements.txt
>> +++ b/tests/requirements.txt
>> @@ -2,5 +2,5 @@
>>  # in the tests/venv Python virtual environment. For more info,
>>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>  # Note that qemu.git/python/ is always implicitly installed.
>> -avocado-framework==88.1
>> +avocado-framework==101.0
>>  pycdlib==1.11.0
>> --
>> 2.39.2
>>
>
> I thought there were test failures that prohibited us from bumping the
> Avocado-Framework version. Did those get rectified recently?

I'm not seeing any with your patch applied.


-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v4 0/6] testing/next: avocado, docs, gitlab

[John Snow]

On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> This is the current state of my testing tree. It contains a few
> patches that didn't make it into 8.0 and more testing fixes including
> a bump to the avocado release including Thomas' --max-parallel-tasks
> hack to avoid running the tests in parallel. So far it doesn't look
> like its causing any problems although I am seeing some timeouts on
> the BSDs.

Oh, looks like you answered my question here. OK then!


>
> The following patches need review:
>
> tests/avocado: use the new snapshots for testing
> tests/requirements.txt: bump up avocado-framework version to 101.0 (0 acks, 1 sobs, 1 tbs)
>
> Alex Bennée (2):
>   tests/avocado: use the new snapshots for testing
>   qemu-options: finesse the recommendations around -blockdev
>
> Kautuk Consul (1):
>   tests/requirements.txt: bump up avocado-framework version to 101.0
>
> Philippe Mathieu-Daudé (1):
>   tests/avocado: Add set of boot tests on SBSA-ref
>
> Thomas Huth (2):
>   gitlab-ci: Avoid to re-run "configure" in the device-crash-test jobs
>   scripts/device-crash-test: Add a parameter to run with TCG only
>
>  MAINTAINERS                              |   1 +
>  .gitlab-ci.d/buildtest.yml               |   6 +-
>  qemu-options.hx                          |  24 +++-
>  scripts/device-crash-test                |   4 +-
>  tests/Makefile.include                   |  18 ++-
>  tests/avocado/machine_aarch64_sbsaref.py | 158 +++++++++++++++++++++
>  tests/avocado/tuxrun_baselines.py        | 170 +++++++++++++++++++----
>  tests/requirements.txt                   |   2 +-
>  8 files changed, 343 insertions(+), 40 deletions(-)
>  create mode 100644 tests/avocado/machine_aarch64_sbsaref.py
>
> --
> 2.39.2
>

Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Thomas Huth]

On 17/04/2023 15.43, Alex Bennée wrote:
> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> 
> Avocado version 101.0 has a fix to re-compute the checksum
> of an asset file if the algorithm used in the *-CHECKSUM
> file isn't the same as the one being passed to it by the
> avocado user (i.e. the avocado_qemu python module).
> In the earlier avocado versions this fix wasn't there due
> to which if the checksum wouldn't match the earlier
> checksum (calculated by a different algorithm), the avocado
> code would start downloading a fresh image from the internet
> URL thus making the test-cases take longer to execute.
> 
> Bump up the avocado-framework version to 101.0.
> 
> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
> 
> ---
> v2
>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> ---
>   tests/Makefile.include | 18 +++++++++++-------
>   tests/requirements.txt |  2 +-
>   2 files changed, 12 insertions(+), 8 deletions(-)
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 9422ddaece..a4de0ad5a2 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>   # download all vm images, according to defined targets
>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>   
> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
> +
>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> -	$(call quiet-command, \
> -            $(TESTS_PYTHON) -m avocado \
> -            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> -			--filter-by-tags-include-empty-key) \
> -            $(AVOCADO_CMDLINE_TAGS) \
> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> +	$(call quiet-command, 							\
> +            $(TESTS_PYTHON) -m avocado 						\
> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) 	\
> +            $(if $(AVOCADO_TAGS),, 						\
> +			--filter-by-tags-include-empty 				\
> +			--filter-by-tags-include-empty-key) 			\
> +		--max-parallel-tasks $(JOBS_OPTION:-j%=%) 			\
> +            $(AVOCADO_CMDLINE_TAGS) 						\
> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), 			\

We might need to revisit this --failfast logic, too. If I've got that right, 
failfast is now the default with the new system? So we might want to disable 
it by default again if GITLAB_CI is not set?

  Thomas

Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Philippe Mathieu-Daudé]

On 17/4/23 19:44, Alex Bennée wrote:
> 
> John Snow <jsnow@redhat.com> writes:
> 
>> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>>>
>>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>>
>>> Avocado version 101.0 has a fix to re-compute the checksum
>>> of an asset file if the algorithm used in the *-CHECKSUM
>>> file isn't the same as the one being passed to it by the
>>> avocado user (i.e. the avocado_qemu python module).
>>> In the earlier avocado versions this fix wasn't there due
>>> to which if the checksum wouldn't match the earlier
>>> checksum (calculated by a different algorithm), the avocado
>>> code would start downloading a fresh image from the internet
>>> URL thus making the test-cases take longer to execute.
>>>
>>> Bump up the avocado-framework version to 101.0.
>>>
>>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>>
>>> ---
>>> v2
>>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>>> ---
>>>   tests/Makefile.include | 18 +++++++++++-------
>>>   tests/requirements.txt |  2 +-
>>>   2 files changed, 12 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index 9422ddaece..a4de0ad5a2 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>>   # download all vm images, according to defined targets
>>>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>>>
>>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
>>> +
>>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>>> -       $(call quiet-command, \
>>> -            $(TESTS_PYTHON) -m avocado \
>>> -            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>>> -                       --filter-by-tags-include-empty-key) \
>>> -            $(AVOCADO_CMDLINE_TAGS) \
>>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>>> +       $(call quiet-command,                                                   \
>>> +            $(TESTS_PYTHON) -m avocado                                                 \
>>> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR)  \
>>> +            $(if $(AVOCADO_TAGS),,                                             \
>>> +                       --filter-by-tags-include-empty                          \
>>> +                       --filter-by-tags-include-empty-key)                     \
>>> +               --max-parallel-tasks $(JOBS_OPTION:-j%=%)                       \
>>> +            $(AVOCADO_CMDLINE_TAGS)                                            \
>>> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),                   \
>>>               "AVOCADO", "tests/avocado")
>>>
>>>   check-acceptance-deprecated-warning:
>>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>>> index 0ba561b6bd..a6f73da681 100644
>>> --- a/tests/requirements.txt
>>> +++ b/tests/requirements.txt
>>> @@ -2,5 +2,5 @@
>>>   # in the tests/venv Python virtual environment. For more info,
>>>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>   # Note that qemu.git/python/ is always implicitly installed.
>>> -avocado-framework==88.1
>>> +avocado-framework==101.0
>>>   pycdlib==1.11.0
>>> --
>>> 2.39.2
>>>
>>
>> I thought there were test failures that prohibited us from bumping the
>> Avocado-Framework version. Did those get rectified recently?

No, still not working on Darwin. At this point I'm thinking at sending
a patch clarifying testing QEMU with Avocado is not supported on Darwin.

> I'm not seeing any with your patch applied.

Are you using running the Cirrus-CI jobs?

Re: [PATCH v4 5/6] scripts/device-crash-test: Add a parameter to run with TCG only

[Philippe Mathieu-Daudé]

On 17/4/23 15:43, Alex Bennée wrote:
> From: Thomas Huth <thuth@redhat.com>
> 
> We're currently facing the problem that the device-crash-test script
> runs twice as long in the CI when a runner supports KVM - which sometimes
> results in a timeout of the CI job. To get a more deterministic runtime
> here, add an option to the script that allows to run it with TCG only.
> 
> Reported-by: Eldon Stegall <eldon-qemu@eldondev.com>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> Message-Id: <20230414145845.456145-3-thuth@redhat.com>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   .gitlab-ci.d/buildtest.yml | 2 +-
>   scripts/device-crash-test  | 4 +++-
>   2 files changed, 4 insertions(+), 2 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 4876 bytes --]

On Thu, Apr 20, 2023, 5:19 AM Philippe Mathieu-Daudé <philmd@linaro.org>
wrote:

> On 17/4/23 19:44, Alex Bennée wrote:
> >
> > John Snow <jsnow@redhat.com> writes:
> >
> >> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org>
> wrote:
> >>>
> >>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> >>>
> >>> Avocado version 101.0 has a fix to re-compute the checksum
> >>> of an asset file if the algorithm used in the *-CHECKSUM
> >>> file isn't the same as the one being passed to it by the
> >>> avocado user (i.e. the avocado_qemu python module).
> >>> In the earlier avocado versions this fix wasn't there due
> >>> to which if the checksum wouldn't match the earlier
> >>> checksum (calculated by a different algorithm), the avocado
> >>> code would start downloading a fresh image from the internet
> >>> URL thus making the test-cases take longer to execute.
> >>>
> >>> Bump up the avocado-framework version to 101.0.
> >>>
> >>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> >>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> >>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> >>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
> >>>
> >>> ---
> >>> v2
> >>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> >>> ---
> >>>   tests/Makefile.include | 18 +++++++++++-------
> >>>   tests/requirements.txt |  2 +-
> >>>   2 files changed, 12 insertions(+), 8 deletions(-)
> >>>
> >>> diff --git a/tests/Makefile.include b/tests/Makefile.include
> >>> index 9422ddaece..a4de0ad5a2 100644
> >>> --- a/tests/Makefile.include
> >>> +++ b/tests/Makefile.include
> >>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
> >>>   # download all vm images, according to defined targets
> >>>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%,
> $(FEDORA_31_DOWNLOAD))
> >>>
> >>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter
> -j%,$(MAKEFLAGS))))
> >>> +
> >>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> >>> -       $(call quiet-command, \
> >>> -            $(TESTS_PYTHON) -m avocado \
> >>> -            --show=$(AVOCADO_SHOW) run
> --job-results-dir=$(TESTS_RESULTS_DIR) \
> >>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> >>> -                       --filter-by-tags-include-empty-key) \
> >>> -            $(AVOCADO_CMDLINE_TAGS) \
> >>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> >>> +       $(call quiet-command,
>          \
> >>> +            $(TESTS_PYTHON) -m avocado
>                  \
> >>> +            --show=$(AVOCADO_SHOW) run
> --job-results-dir=$(TESTS_RESULTS_DIR)  \
> >>> +            $(if $(AVOCADO_TAGS),,
>          \
> >>> +                       --filter-by-tags-include-empty
>           \
> >>> +                       --filter-by-tags-include-empty-key)
>          \
> >>> +               --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>          \
> >>> +            $(AVOCADO_CMDLINE_TAGS)
>           \
> >>> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),
>          \
> >>>               "AVOCADO", "tests/avocado")
> >>>
> >>>   check-acceptance-deprecated-warning:
> >>> diff --git a/tests/requirements.txt b/tests/requirements.txt
> >>> index 0ba561b6bd..a6f73da681 100644
> >>> --- a/tests/requirements.txt
> >>> +++ b/tests/requirements.txt
> >>> @@ -2,5 +2,5 @@
> >>>   # in the tests/venv Python virtual environment. For more info,
> >>>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> >>>   # Note that qemu.git/python/ is always implicitly installed.
> >>> -avocado-framework==88.1
> >>> +avocado-framework==101.0
> >>>   pycdlib==1.11.0
> >>> --
> >>> 2.39.2
> >>>
> >>
> >> I thought there were test failures that prohibited us from bumping the
> >> Avocado-Framework version. Did those get rectified recently?
>
> No, still not working on Darwin. At this point I'm thinking at sending
> a patch clarifying testing QEMU with Avocado is not supported on Darwin.
>

What's broken on Darwin?

Is it broken with avocado-framework==88.1 too? (Is this a regression?)

We might need to consider entirely separate issues:

(1) We need to upgrade avocado-framework to a supported LTS version >= v90
for dependency harmony with qemu.git/python testing

(2) We need to upgrade avocado-framework to >= v101.0 to fix the stated
checksum issue in this patch

(3) We would like avocado tests to work on Darwin. (Have they ever worked?
When did they break? etc.)

My understanding is that this patch helps with 1 and 2 but not 3, but if
Darwin support isn't a regression, this is still a step forward.


> > I'm not seeing any with your patch applied.
>
> Are you using running the Cirrus-CI jobs?
>
>

[-- Attachment #2: Type: text/html, Size: 7748 bytes --]

Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Cleber Rosa]

On 4/17/23 09:43, Alex Bennée wrote:
> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>
> Avocado version 101.0 has a fix to re-compute the checksum
> of an asset file if the algorithm used in the *-CHECKSUM
> file isn't the same as the one being passed to it by the
> avocado user (i.e. the avocado_qemu python module).
> In the earlier avocado versions this fix wasn't there due
> to which if the checksum wouldn't match the earlier
> checksum (calculated by a different algorithm), the avocado
> code would start downloading a fresh image from the internet
> URL thus making the test-cases take longer to execute.
>
> Bump up the avocado-framework version to 101.0.
>
> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>
> ---
> v2
>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> ---
>   tests/Makefile.include | 18 +++++++++++-------
>   tests/requirements.txt |  2 +-
>   2 files changed, 12 insertions(+), 8 deletions(-)

Hi everyone,

Looks like I've mistakenly replied to the earlier thread, so here's 
pretty much the same message on a better location.

First of all, thanks to Kautuk for sending this and thanks to Hariharan 
for further testing it.

I'd like to give some context which not everyone may be aware of. 
Avocado 101.0 is a very different when compared with 88.1. Everything 
related to the execution of tests is brand new.  To be more precise, on 
version 91.0[1], this new runner[2] became the default. On version 97.0, 
the old runner implementation (currently in use in QEMU) was finally 
removed.

On most releases since then, I've been running the QEMU tests with the 
latest Avocado, and finding issues that are (as resources allow) 
addressed in later versions.   As you probably noticed, Avocado 101.0 
runs the QEMU tests without much (or any) visible issues for most 
people.  But, I'm aware of two pending issues that may or may not be a 
big deal to users:

I) The logging behavior is a bit different since Avocado 88.1. At a 
given point it was considered that Avocado should not mess around 
inadvertently with Python's root logger, and should be more picky about 
it includes in logs.  For most cases, a simple workaround[4] does the 
trick.  But, for some other use cases (say for 3rd party libraries' logs 
you want logged alongside Avocado's logs) there's a pending PR[5] that 
will take care of all known limitations.

II) The support for killing tests (internally in Avocado represented as 
more generic "tasks") and all its children is a bit lacking.  This is an 
issue I'm actively working on[6].  This may leave some processes (such 
as "qemu-system-*") running even after a test was interrupted.

Fixes for both of these issues are due to be in version 102.0. The ETA 
for version 102.0 is 1-2 weeks.

With that being said, I'm more than OK with this patch (alongside PATCH 
2, without which havoc ensues :) provided people understand the two 
pending issues above.  If this patch is taken before Avocado 102.0 is 
released, the delta from 101.0 would be much smaller, so it should be an 
easier change to test.

Cheers,

- Cleber.


[1] - https://avocado-framework.readthedocs.io/en/101.0/releases/91_0.html

[2] - The new runner is called "nrunner" and I am to be blamed for the 
naming lacking any originality

[3] - 
https://avocado-framework.readthedocs.io/en/101.0/releases/97_0.html#users-test-writers

[4] - 
https://gitlab.com/cleber.gnu/qemu/-/commit/a9f39c4f6671b756196a185c7275eb7ebd13e588

[5] - https://github.com/avocado-framework/avocado/pull/5645

[6] - https://github.com/avocado-framework/avocado/issues/4994

Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Cleber Rosa]

On 4/17/23 12:50, John Snow wrote:
> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>
>> Avocado version 101.0 has a fix to re-compute the checksum
>> of an asset file if the algorithm used in the *-CHECKSUM
>> file isn't the same as the one being passed to it by the
>> avocado user (i.e. the avocado_qemu python module).
>> In the earlier avocado versions this fix wasn't there due
>> to which if the checksum wouldn't match the earlier
>> checksum (calculated by a different algorithm), the avocado
>> code would start downloading a fresh image from the internet
>> URL thus making the test-cases take longer to execute.
>>
>> Bump up the avocado-framework version to 101.0.
>>
>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>
>> ---
>> v2
>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>> ---
>>   tests/Makefile.include | 18 +++++++++++-------
>>   tests/requirements.txt |  2 +-
>>   2 files changed, 12 insertions(+), 8 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 9422ddaece..a4de0ad5a2 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>   # download all vm images, according to defined targets
>>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>>
>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
>> +
>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>> -       $(call quiet-command, \
>> -            $(TESTS_PYTHON) -m avocado \
>> -            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>> -                       --filter-by-tags-include-empty-key) \
>> -            $(AVOCADO_CMDLINE_TAGS) \
>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>> +       $(call quiet-command,                                                   \
>> +            $(TESTS_PYTHON) -m avocado                                                 \
>> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR)  \
>> +            $(if $(AVOCADO_TAGS),,                                             \
>> +                       --filter-by-tags-include-empty                          \
>> +                       --filter-by-tags-include-empty-key)                     \
>> +               --max-parallel-tasks $(JOBS_OPTION:-j%=%)                       \
>> +            $(AVOCADO_CMDLINE_TAGS)                                            \
>> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),                   \
>>               "AVOCADO", "tests/avocado")
>>
>>   check-acceptance-deprecated-warning:
>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>> index 0ba561b6bd..a6f73da681 100644
>> --- a/tests/requirements.txt
>> +++ b/tests/requirements.txt
>> @@ -2,5 +2,5 @@
>>   # in the tests/venv Python virtual environment. For more info,
>>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>   # Note that qemu.git/python/ is always implicitly installed.
>> -avocado-framework==88.1
>> +avocado-framework==101.0
>>   pycdlib==1.11.0
>> --
>> 2.39.2
>>
> I thought there were test failures that prohibited us from bumping the
> Avocado-Framework version. Did those get rectified recently?
>
Hi John,

While testing before and after behavior of pretty much every test in 
tests/avocado/,  I've found that some tests may get affected by the 
extra isolation (and overhead) of the new runner in recent Avocado and 
may get interrupted more often than on the previous runner.  But this is 
highly dependent on the amount of resources the machine you run the 
tests on has, and how tight the current timeout is.

Running all the tests under old and new Avocado also revealed some tests 
that are equally broken under both versions.  Thomas has bisected some, 
and Cedric has come up with some patches too.

To risk not being too verbose by default, let me know if you want the 
full details.  Also, see my previous reply about the two issues I'm 
aware that would make this bump as seamless as possible.

Thanks,

- Cleber.

Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Cleber Rosa]

On 4/20/23 05:18, Philippe Mathieu-Daudé wrote:
> On 17/4/23 19:44, Alex Bennée wrote:
>>
>> John Snow <jsnow@redhat.com> writes:
>>
>>> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> 
>>> wrote:
>>>>
>>>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>>>
>>>> Avocado version 101.0 has a fix to re-compute the checksum
>>>> of an asset file if the algorithm used in the *-CHECKSUM
>>>> file isn't the same as the one being passed to it by the
>>>> avocado user (i.e. the avocado_qemu python module).
>>>> In the earlier avocado versions this fix wasn't there due
>>>> to which if the checksum wouldn't match the earlier
>>>> checksum (calculated by a different algorithm), the avocado
>>>> code would start downloading a fresh image from the internet
>>>> URL thus making the test-cases take longer to execute.
>>>>
>>>> Bump up the avocado-framework version to 101.0.
>>>>
>>>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>>>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>>>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>>>
>>>> ---
>>>> v2
>>>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>>>> ---
>>>>   tests/Makefile.include | 18 +++++++++++-------
>>>>   tests/requirements.txt |  2 +-
>>>>   2 files changed, 12 insertions(+), 8 deletions(-)
>>>>
>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>> index 9422ddaece..a4de0ad5a2 100644
>>>> --- a/tests/Makefile.include
>>>> +++ b/tests/Makefile.include
>>>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>>>   # download all vm images, according to defined targets
>>>>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, 
>>>> $(FEDORA_31_DOWNLOAD))
>>>>
>>>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter 
>>>> -j%,$(MAKEFLAGS))))
>>>> +
>>>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>>>> -       $(call quiet-command, \
>>>> -            $(TESTS_PYTHON) -m avocado \
>>>> -            --show=$(AVOCADO_SHOW) run 
>>>> --job-results-dir=$(TESTS_RESULTS_DIR) \
>>>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>>>> -                       --filter-by-tags-include-empty-key) \
>>>> -            $(AVOCADO_CMDLINE_TAGS) \
>>>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>>>> +       $(call quiet-command, \
>>>> +            $(TESTS_PYTHON) -m 
>>>> avocado                                                 \
>>>> +            --show=$(AVOCADO_SHOW) run 
>>>> --job-results-dir=$(TESTS_RESULTS_DIR)  \
>>>> +            $(if $(AVOCADO_TAGS),, \
>>>> + --filter-by-tags-include-empty                          \
>>>> + --filter-by-tags-include-empty-key)                     \
>>>> +               --max-parallel-tasks 
>>>> $(JOBS_OPTION:-j%=%)                       \
>>>> + $(AVOCADO_CMDLINE_TAGS) \
>>>> +            $(if $(GITLAB_CI),,--failfast) 
>>>> $(AVOCADO_TESTS),                   \
>>>>               "AVOCADO", "tests/avocado")
>>>>
>>>>   check-acceptance-deprecated-warning:
>>>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>>>> index 0ba561b6bd..a6f73da681 100644
>>>> --- a/tests/requirements.txt
>>>> +++ b/tests/requirements.txt
>>>> @@ -2,5 +2,5 @@
>>>>   # in the tests/venv Python virtual environment. For more info,
>>>>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>>   # Note that qemu.git/python/ is always implicitly installed.
>>>> -avocado-framework==88.1
>>>> +avocado-framework==101.0
>>>>   pycdlib==1.11.0
>>>> -- 
>>>> 2.39.2
>>>>
>>>
>>> I thought there were test failures that prohibited us from bumping the
>>> Avocado-Framework version. Did those get rectified recently?
>
> No, still not working on Darwin. At this point I'm thinking at sending
> a patch clarifying testing QEMU with Avocado is not supported on Darwin.
>
Hi Phil,

Do you mean with or without this version bump?

Even though it was somewhat recently[1] that another level of Darwin 
compatibility was added to Avocado, where a good part of Avocado's 
selftests[2] were enabled in OS X, Avocado 101.0 should not be that far 
behind.

I honestly believe people testing and reporting successful execution of 
the Avocado tests under Darwin with latest Avocado. Hopefully my memory 
is not playing tricks on me, but I'll attempt to verify that.

Cheers,

- Cleber.

[1] - https://github.com/avocado-framework/avocado/pull/5622

[2] - 
https://github.com/avocado-framework/avocado/actions/runs/4417285317/jobs/7742772932?pr=5622#step:10:11


>> I'm not seeing any with your patch applied.
>
> Are you using running the Cirrus-CI jobs?
>

Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Cleber Rosa]

On 4/20/23 12:54, John Snow wrote:
>
>
> On Thu, Apr 20, 2023, 5:19 AM Philippe Mathieu-Daudé 
> <philmd@linaro.org> wrote:
>
>     On 17/4/23 19:44, Alex Bennée wrote:
>     >
>     > John Snow <jsnow@redhat.com> writes:
>     >
>     >> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée
>     <alex.bennee@linaro.org> wrote:
>     >>>
>     >>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>     >>>
>     >>> Avocado version 101.0 has a fix to re-compute the checksum
>     >>> of an asset file if the algorithm used in the *-CHECKSUM
>     >>> file isn't the same as the one being passed to it by the
>     >>> avocado user (i.e. the avocado_qemu python module).
>     >>> In the earlier avocado versions this fix wasn't there due
>     >>> to which if the checksum wouldn't match the earlier
>     >>> checksum (calculated by a different algorithm), the avocado
>     >>> code would start downloading a fresh image from the internet
>     >>> URL thus making the test-cases take longer to execute.
>     >>>
>     >>> Bump up the avocado-framework version to 101.0.
>     >>>
>     >>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>     >>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>     >>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>     >>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>     >>>
>     >>> ---
>     >>> v2
>     >>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>     >>> ---
>     >>>   tests/Makefile.include | 18 +++++++++++-------
>     >>>   tests/requirements.txt |  2 +-
>     >>>   2 files changed, 12 insertions(+), 8 deletions(-)
>     >>>
>     >>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>     >>> index 9422ddaece..a4de0ad5a2 100644
>     >>> --- a/tests/Makefile.include
>     >>> +++ b/tests/Makefile.include
>     >>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>     >>>   # download all vm images, according to defined targets
>     >>>   get-vm-images: check-venv $(patsubst
>     %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>     >>>
>     >>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter
>     -j%,$(MAKEFLAGS))))
>     >>> +
>     >>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>     >>> -       $(call quiet-command, \
>     >>> -            $(TESTS_PYTHON) -m avocado \
>     >>> -            --show=$(AVOCADO_SHOW) run
>     --job-results-dir=$(TESTS_RESULTS_DIR) \
>     >>> -            $(if $(AVOCADO_TAGS),,
>     --filter-by-tags-include-empty \
>     >>> -  --filter-by-tags-include-empty-key) \
>     >>> -            $(AVOCADO_CMDLINE_TAGS) \
>     >>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>     >>> +       $(call quiet-command,                                    \
>     >>> +            $(TESTS_PYTHON) -m avocado                      
>                          \
>     >>> +            --show=$(AVOCADO_SHOW) run
>     --job-results-dir=$(TESTS_RESULTS_DIR)  \
>     >>> +            $(if $(AVOCADO_TAGS),,                          
>              \
>     >>> +  --filter-by-tags-include-empty                          \
>     >>> +  --filter-by-tags-include-empty-key)                     \
>     >>> +               --max-parallel-tasks $(JOBS_OPTION:-j%=%)     
>                      \
>     >>> +            $(AVOCADO_CMDLINE_TAGS)                          
>             \
>     >>> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), 
>                      \
>     >>>               "AVOCADO", "tests/avocado")
>     >>>
>     >>>   check-acceptance-deprecated-warning:
>     >>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>     >>> index 0ba561b6bd..a6f73da681 100644
>     >>> --- a/tests/requirements.txt
>     >>> +++ b/tests/requirements.txt
>     >>> @@ -2,5 +2,5 @@
>     >>>   # in the tests/venv Python virtual environment. For more info,
>     >>>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>     >>>   # Note that qemu.git/python/ is always implicitly installed.
>     >>> -avocado-framework==88.1
>     >>> +avocado-framework==101.0
>     >>>   pycdlib==1.11.0
>     >>> --
>     >>> 2.39.2
>     >>>
>     >>
>     >> I thought there were test failures that prohibited us from
>     bumping the
>     >> Avocado-Framework version. Did those get rectified recently?
>
>     No, still not working on Darwin. At this point I'm thinking at sending
>     a patch clarifying testing QEMU with Avocado is not supported on
>     Darwin.
>
>
> What's broken on Darwin?
>
Hi John,

IIRC, the main issue is that the legacy runner (the one active in 88.1) 
pickles some stuff that do not play nice under Darwin.  It has never (to 
the best of my knowledge) worked under Darwin.

> Is it broken with avocado-framework==88.1 too? (Is this a regression?)
>
It was already broken with avocado-framework==88.1, but it should mostly 
work with 101.0 (see previous reply).

> We might need to consider entirely separate issues:
>
> (1) We need to upgrade avocado-framework to a supported LTS version >= 
> v90 for dependency harmony with qemu.git/python testing
>
> (2) We need to upgrade avocado-framework to >= v101.0 to fix the 
> stated checksum issue in this patch
>
> (3) We would like avocado tests to work on Darwin. (Have they ever 
> worked? When did they break? etc.)
>
> My understanding is that this patch helps with 1 and 2 but not 3, but 
> if Darwin support isn't a regression, this is still a step forward.
>
To the best of my knowledge, this patch would help with 1, 2 and even 3.

Cheers,

- Cleber.

>
>     > I'm not seeing any with your patch applied.
>
>     Are you using running the Cirrus-CI jobs?
>

Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Cleber Rosa]

On 4/18/23 03:10, Thomas Huth wrote:
> On 17/04/2023 15.43, Alex Bennée wrote:
>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>
>> Avocado version 101.0 has a fix to re-compute the checksum
>> of an asset file if the algorithm used in the *-CHECKSUM
>> file isn't the same as the one being passed to it by the
>> avocado user (i.e. the avocado_qemu python module).
>> In the earlier avocado versions this fix wasn't there due
>> to which if the checksum wouldn't match the earlier
>> checksum (calculated by a different algorithm), the avocado
>> code would start downloading a fresh image from the internet
>> URL thus making the test-cases take longer to execute.
>>
>> Bump up the avocado-framework version to 101.0.
>>
>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>
>> ---
>> v2
>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>> ---
>>   tests/Makefile.include | 18 +++++++++++-------
>>   tests/requirements.txt |  2 +-
>>   2 files changed, 12 insertions(+), 8 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 9422ddaece..a4de0ad5a2 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>   # download all vm images, according to defined targets
>>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, 
>> $(FEDORA_31_DOWNLOAD))
>>   +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter 
>> -j%,$(MAKEFLAGS))))
>> +
>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>> -    $(call quiet-command, \
>> -            $(TESTS_PYTHON) -m avocado \
>> -            --show=$(AVOCADO_SHOW) run 
>> --job-results-dir=$(TESTS_RESULTS_DIR) \
>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>> -            --filter-by-tags-include-empty-key) \
>> -            $(AVOCADO_CMDLINE_TAGS) \
>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>> +    $(call quiet-command,                             \
>> +            $(TESTS_PYTHON) -m avocado \
>> +            --show=$(AVOCADO_SHOW) run 
>> --job-results-dir=$(TESTS_RESULTS_DIR)     \
>> +            $(if $(AVOCADO_TAGS),,                         \
>> +            --filter-by-tags-include-empty                 \
>> +            --filter-by-tags-include-empty-key)             \
>> +        --max-parallel-tasks $(JOBS_OPTION:-j%=%)             \
>> +            $(AVOCADO_CMDLINE_TAGS)                         \
>> +            $(if $(GITLAB_CI),,--failfast) 
>> $(AVOCADO_TESTS),             \
>
> We might need to revisit this --failfast logic, too. If I've got that 
> right, failfast is now the default with the new system? So we might 
> want to disable it by default again if GITLAB_CI is not set?
>
>  Thomas
>
Hi Thomas,

I must be missing something, because under Avocado 101.0, I'm getting 
the following behavior without the --failfast flag:

   $ avocado run --max-parallel-tasks=1 -- /bin/true /bin/false /bin/true
   JOB ID     : 646f476f01b8d5599a57530606de543f2d9a5366
   JOB LOG    : 
/root/avocado/job-results/job-2023-04-21T04.15-646f476/job.log
    (1/3) /bin/true: STARTED
    (1/3) /bin/true: PASS (0.01 s)
    (2/3) /bin/false: STARTED
    (2/3) /bin/false: FAIL (0.01 s)
    (3/3) /bin/true: STARTED
    (3/3) /bin/true: PASS (0.01 s)
   RESULTS    : PASS 2 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 
0 | CANCEL 0
   JOB TIME   : 1.01 s

   Test summary:
   /bin/false: FAIL

And this with --failfast:

   $ avocado run --failfast --max-parallel-tasks=1 -- /bin/true 
/bin/false /bin/true
   JOB ID     : ae4894607a42194a7382efa545eccaccf7495fa3
   JOB LOG    : 
/root/avocado/job-results/job-2023-04-21T04.17-ae48946/job.log
    (1/3) /bin/true: STARTED
    (1/3) /bin/true: PASS (0.01 s)
    (2/3) /bin/false: STARTED
    (2/3) /bin/false: FAIL (0.01 s)
   Interrupting job (failfast).
   RESULTS    : PASS 1 | ERROR 0 | FAIL 1 | SKIP 1 | WARN 0 | INTERRUPT 
0 | CANCEL 0
   JOB TIME   : 0.75 s

   Test summary:
   /bin/false: FAIL

Maybe it's something in the Makefile I'm missing, or something specific 
to these tests, but that would be a long shot.

Thanks,

- Cleber.

Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Thomas Huth]

On 21/04/2023 06.19, Cleber Rosa wrote:
> 
> On 4/18/23 03:10, Thomas Huth wrote:
>> On 17/04/2023 15.43, Alex Bennée wrote:
>>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>>
>>> Avocado version 101.0 has a fix to re-compute the checksum
>>> of an asset file if the algorithm used in the *-CHECKSUM
>>> file isn't the same as the one being passed to it by the
>>> avocado user (i.e. the avocado_qemu python module).
>>> In the earlier avocado versions this fix wasn't there due
>>> to which if the checksum wouldn't match the earlier
>>> checksum (calculated by a different algorithm), the avocado
>>> code would start downloading a fresh image from the internet
>>> URL thus making the test-cases take longer to execute.
>>>
>>> Bump up the avocado-framework version to 101.0.
>>>
>>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>>
>>> ---
>>> v2
>>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>>> ---
>>>   tests/Makefile.include | 18 +++++++++++-------
>>>   tests/requirements.txt |  2 +-
>>>   2 files changed, 12 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index 9422ddaece..a4de0ad5a2 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>>   # download all vm images, according to defined targets
>>>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, 
>>> $(FEDORA_31_DOWNLOAD))
>>>   +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
>>> +
>>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>>> -    $(call quiet-command, \
>>> -            $(TESTS_PYTHON) -m avocado \
>>> -            --show=$(AVOCADO_SHOW) run 
>>> --job-results-dir=$(TESTS_RESULTS_DIR) \
>>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>>> -            --filter-by-tags-include-empty-key) \
>>> -            $(AVOCADO_CMDLINE_TAGS) \
>>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>>> +    $(call quiet-command,                             \
>>> +            $(TESTS_PYTHON) -m avocado \
>>> +            --show=$(AVOCADO_SHOW) run 
>>> --job-results-dir=$(TESTS_RESULTS_DIR)     \
>>> +            $(if $(AVOCADO_TAGS),,                         \
>>> +            --filter-by-tags-include-empty                 \
>>> +            --filter-by-tags-include-empty-key)             \
>>> +        --max-parallel-tasks $(JOBS_OPTION:-j%=%)             \
>>> +            $(AVOCADO_CMDLINE_TAGS)                         \
>>> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),             \
>>
>> We might need to revisit this --failfast logic, too. If I've got that 
>> right, failfast is now the default with the new system? So we might want 
>> to disable it by default again if GITLAB_CI is not set?
>>
>>  Thomas
>>
> Hi Thomas,
> 
> I must be missing something, because under Avocado 101.0, I'm getting the 
> following behavior without the --failfast flag:
> 
>    $ avocado run --max-parallel-tasks=1 -- /bin/true /bin/false /bin/true
>    JOB ID     : 646f476f01b8d5599a57530606de543f2d9a5366
>    JOB LOG    : /root/avocado/job-results/job-2023-04-21T04.15-646f476/job.log
>     (1/3) /bin/true: STARTED
>     (1/3) /bin/true: PASS (0.01 s)
>     (2/3) /bin/false: STARTED
>     (2/3) /bin/false: FAIL (0.01 s)
>     (3/3) /bin/true: STARTED
>     (3/3) /bin/true: PASS (0.01 s)
>    RESULTS    : PASS 2 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
> CANCEL 0
>    JOB TIME   : 1.01 s
> 
>    Test summary:
>    /bin/false: FAIL
> 
> And this with --failfast:
> 
>    $ avocado run --failfast --max-parallel-tasks=1 -- /bin/true /bin/false 
> /bin/true
>    JOB ID     : ae4894607a42194a7382efa545eccaccf7495fa3
>    JOB LOG    : /root/avocado/job-results/job-2023-04-21T04.17-ae48946/job.log
>     (1/3) /bin/true: STARTED
>     (1/3) /bin/true: PASS (0.01 s)
>     (2/3) /bin/false: STARTED
>     (2/3) /bin/false: FAIL (0.01 s)
>    Interrupting job (failfast).
>    RESULTS    : PASS 1 | ERROR 0 | FAIL 1 | SKIP 1 | WARN 0 | INTERRUPT 0 | 
> CANCEL 0
>    JOB TIME   : 0.75 s
> 
>    Test summary:
>    /bin/false: FAIL
> 
> Maybe it's something in the Makefile I'm missing, or something specific to 
> these tests, but that would be a long shot.

Ah, thanks, I think I simply got it mixed up while trying to read the above 
Makefile logic - never mind my question, it should be fine.

  Thomas

Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Thomas Huth]

On 17/04/2023 15.43, Alex Bennée wrote:
> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> 
> Avocado version 101.0 has a fix to re-compute the checksum
> of an asset file if the algorithm used in the *-CHECKSUM
> file isn't the same as the one being passed to it by the
> avocado user (i.e. the avocado_qemu python module).
> In the earlier avocado versions this fix wasn't there due
> to which if the checksum wouldn't match the earlier
> checksum (calculated by a different algorithm), the avocado
> code would start downloading a fresh image from the internet
> URL thus making the test-cases take longer to execute.
> 
> Bump up the avocado-framework version to 101.0.
> 
> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
> 
> ---
> v2
>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> ---
>   tests/Makefile.include | 18 +++++++++++-------
>   tests/requirements.txt |  2 +-
>   2 files changed, 12 insertions(+), 8 deletions(-)
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 9422ddaece..a4de0ad5a2 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>   # download all vm images, according to defined targets
>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>   
> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
> +
>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> -	$(call quiet-command, \
> -            $(TESTS_PYTHON) -m avocado \
> -            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> -			--filter-by-tags-include-empty-key) \
> -            $(AVOCADO_CMDLINE_TAGS) \
> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> +	$(call quiet-command, 							\
> +            $(TESTS_PYTHON) -m avocado 						\
> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) 	\
> +            $(if $(AVOCADO_TAGS),, 						\
> +			--filter-by-tags-include-empty 				\
> +			--filter-by-tags-include-empty-key) 			\
> +		--max-parallel-tasks $(JOBS_OPTION:-j%=%) 			\
> +            $(AVOCADO_CMDLINE_TAGS) 						\
> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), 			\
>               "AVOCADO", "tests/avocado")
>   
>   check-acceptance-deprecated-warning:
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index 0ba561b6bd..a6f73da681 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -2,5 +2,5 @@
>   # in the tests/venv Python virtual environment. For more info,
>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>   # Note that qemu.git/python/ is always implicitly installed.
> -avocado-framework==88.1
> +avocado-framework==101.0
>   pycdlib==1.11.0

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Philippe Mathieu-Daudé]

On 21/4/23 06:09, Cleber Rosa wrote:
> 
> On 4/20/23 12:54, John Snow wrote:
>>
>>
>> On Thu, Apr 20, 2023, 5:19 AM Philippe Mathieu-Daudé 
>> <philmd@linaro.org> wrote:
>>
>>     On 17/4/23 19:44, Alex Bennée wrote:
>>     >
>>     > John Snow <jsnow@redhat.com> writes:
>>     >
>>     >> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée
>>     <alex.bennee@linaro.org> wrote:

>>     >>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>>     >>> index 0ba561b6bd..a6f73da681 100644
>>     >>> --- a/tests/requirements.txt
>>     >>> +++ b/tests/requirements.txt
>>     >>> @@ -2,5 +2,5 @@
>>     >>>   # in the tests/venv Python virtual environment. For more info,
>>     >>>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>     >>>   # Note that qemu.git/python/ is always implicitly installed.
>>     >>> -avocado-framework==88.1
>>     >>> +avocado-framework==101.0
>>     >>>   pycdlib==1.11.0
>>     >>> --
>>     >>> 2.39.2
>>     >>>
>>     >>
>>     >> I thought there were test failures that prohibited us from
>>     bumping the
>>     >> Avocado-Framework version. Did those get rectified recently?
>>
>>     No, still not working on Darwin. At this point I'm thinking at 
>> sending
>>     a patch clarifying testing QEMU with Avocado is not supported on
>>     Darwin.
>>
>>
>> What's broken on Darwin?
>>
> Hi John,
> 
> IIRC, the main issue is that the legacy runner (the one active in 88.1) 
> pickles some stuff that do not play nice under Darwin.  It has never (to 
> the best of my knowledge) worked under Darwin.

Correct.

>> Is it broken with avocado-framework==88.1 too? (Is this a regression?)
>>
> It was already broken with avocado-framework==88.1,

Correct.

> but it should mostly 
> work with 101.0 (see previous reply).
> 
>> We might need to consider entirely separate issues:
>>
>> (1) We need to upgrade avocado-framework to a supported LTS version >= 
>> v90 for dependency harmony with qemu.git/python testing
>>
>> (2) We need to upgrade avocado-framework to >= v101.0 to fix the 
>> stated checksum issue in this patch
>>
>> (3) We would like avocado tests to work on Darwin. (Have they ever 
>> worked? When did they break? etc.)
>>
>> My understanding is that this patch helps with 1 and 2 but not 3, but 
>> if Darwin support isn't a regression, this is still a step forward.
>>
> To the best of my knowledge, this patch would help with 1, 2 and even 3.

As of today using v8.0.0:

$ make check-avocado
   GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 
tests/fp/berkeley-softfloat-3 dtc
   MKDIR   /Users/philmd/source/qemu/build/tests/results
   AVOCADO Downloading avocado tests VM image for aarch64
The image was downloaded:
Provider Version Architecture File
fedora   31      aarch64 
/Users/philmd/avocado/data/cache/by_location/4f156e531446a679cbfe13caef8b7c9f9f79aafa/Fedora-Cloud-Base-31-1.9.aarch64.qcow2
   AVOCADO tests/avocado
Fetching asset from 
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf
Fetching asset from 
tests/avocado/boot_xen.py:BootXen.test_arm64_xen_411_and_dom0
Fetching asset from 
tests/avocado/boot_xen.py:BootXen.test_arm64_xen_414_and_dom0
Fetching asset from 
tests/avocado/boot_xen.py:BootXen.test_arm64_xen_415_and_dom0
Fetching asset from 
tests/avocado/machine_aarch64_virt.py:Aarch64VirtMachine.test_alpine_virt_tcg_gic_max
Fetching asset from 
tests/avocado/replay_kernel.py:ReplayKernelNormal.test_aarch64_virt
Fetching asset from 
tests/avocado/reverse_debugging.py:ReverseDebugging_AArch64.test_aarch64_virt
JOB ID     : 142a0b2d3960c7daccd657781d52f814e1617392
JOB LOG    : 
/Users/philmd/source/qemu/build/tests/results/job-2023-04-21T11.35-142a0b2/job.log

Avocado crashed: TypeError: cannot pickle '_thread.RLock' object
Traceback (most recent call last):

   File 
"/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/core/job.py", 
line 632, in run_tests
     summary |= suite.run(self)

   File 
"/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/core/suite.py", 
line 311, in run
     return self.runner.run_suite(job, self)

   File 
"/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/plugins/runner.py", 
line 390, in run_suite
     if not self.run_test(job, test_factory, queue, summary,

   File 
"/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/plugins/runner.py", 
line 170, in run_test
     proc.start()

   File 
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", 
line 121, in start
     self._popen = self._Popen(self)

   File 
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", 
line 224, in _Popen
     return _default_context.get_context().Process._Popen(process_obj)

   File 
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", 
line 288, in _Popen
     return Popen(process_obj)

   File 
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", 
line 32, in __init__
     super().__init__(process_obj)

   File 
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_fork.py", 
line 19, in __init__
     self._launch(process_obj)

   File 
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", 
line 47, in _launch
     reduction.dump(process_obj, fp)

   File 
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/reduction.py", 
line 60, in dump
     ForkingPickler(file, protocol).dump(obj)

TypeError: cannot pickle '_thread.RLock' object

Please include the traceback info and command line used on your bug report
Report bugs visiting https://github.com/avocado-framework/avocado/issues/new
make: *** [check-avocado] Error 4

This is https://github.com/avocado-framework/avocado/issues/4888
which is still opened.

Manually using "avocado run --test-runner=nrunner" helps running the
tests but the console output is never displayed (see [*]) and once
finished Avocado lets various QEMU processes running in detached
mode, wasting resources.

[*] 
https://lore.kernel.org/qemu-devel/1d0baf8b-c757-265c-b206-07ca3f218b2a@linaro.org/

Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Philippe Mathieu-Daudé]

On 21/4/23 12:07, Philippe Mathieu-Daudé wrote:
> On 21/4/23 06:09, Cleber Rosa wrote:

> Manually using "avocado run --test-runner=nrunner" helps running the
> tests but the console output is never displayed (see [*]) and once
> finished Avocado lets various QEMU processes running in detached
> mode, wasting resources.=

Using this patch:

$ ./tests/venv/bin/avocado --show=console -V run -t arch:aarch64 
tests/avocado/boot_*
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf: 
ERROR
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_xlnx_versal_virt: 
ERROR

Not asking for the console output at least show the test being
processed (similar to old runner):

$ ./tests/venv/bin/avocado run -t machine:raspi3b tests/avocado/boot_*
Fetching asset from 
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf
JOB ID     : 1c963c12f7f07e109ef8480ddf204f02e61f8b8d
JOB LOG    : 
/Users/philmd/avocado/job-results/job-2023-04-21T12.24-1c963c1/job.log
  (1/1) 
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf: 
STARTED
  (1/1) 
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf: 
ERROR: ConnectError: Failed to establish session: EOFError\n	Exit code: 
1\n	Command: ./qemu-system-aarch64 -display none -vga none -chardev 
socket,id=mon,fd=11 -mon chardev=mon,mode=control -machine raspi3b 
-serial null -chardev socket,id=console,path=/var/folders... (0.31 s)
RESULTS    : PASS 0 | ERROR 1 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
CANCEL 0
JOB TIME   : 2.20 s

Test summary:
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf: 
ERROR