[PATCH v2 0/7] final fixes for 8.2

[PATCH v2 0/7] final fixes for 8.2

[Alex Bennée]

8.2 is looking fairly stable but I do have one bug fix for gdbstub
which I came across while debugging something else. The changes for
avocado rationalise all flaky tests under the QEMU_TEST_FLAKY_TESTS
environment variable. The final patch re-adds the flaky tests to the
CI as a manually run allow_fail job so we can still attempt to debug
their failure in the place they tend to fall over.

v2
--

  - addressed some review comments
  - emphasised raising a bug for failing tests
  - drop some tests now missing assets online
  - fix wrong microblaze target for tests

The following still need review:

  gitlab: build the correct microblaze target
  tests/avocado: tag sbsa tests as tcg only
  tests/avocado: drop malta yamon tests
  tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test

Alex.

Alex Bennée (7):
  gdbstub: use a better signal when we halt for IO reasons
  docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS
  tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test
  tests/avocado: drop malta yamon tests
  tests/avocado: tag sbsa tests as tcg only
  gitlab: build the correct microblaze target
  gitlab: add optional job to run flaky avocado tests

 docs/devel/testing.rst                   | 32 ++++++++++------
 gdbstub/internals.h                      |  1 +
 gdbstub/system.c                         |  2 +-
 .gitlab-ci.d/buildtest.yml               | 32 +++++++++++++++-
 tests/avocado/boot_linux.py              | 10 +++--
 tests/avocado/boot_linux_console.py      | 28 ++------------
 tests/avocado/intel_iommu.py             |  6 ++-
 tests/avocado/linux_initrd.py            |  7 +++-
 tests/avocado/machine_aarch64_sbsaref.py |  1 +
 tests/avocado/machine_aspeed.py          | 10 +++--
 tests/avocado/machine_mips_malta.py      | 47 ++++--------------------
 tests/avocado/machine_rx_gdbsim.py       | 10 +++--
 tests/avocado/machine_s390_ccw_virtio.py |  3 +-
 tests/avocado/replay_kernel.py           |  8 +++-
 tests/avocado/reverse_debugging.py       | 16 +++++---
 tests/avocado/smmu.py                    |  6 ++-
 tests/avocado/tuxrun_baselines.py        |  5 ++-
 17 files changed, 122 insertions(+), 102 deletions(-)

-- 
2.39.2

[PATCH v2 1/7] gdbstub: use a better signal when we halt for IO reasons

[Alex Bennée]

The gdb description GDB_SIGNAL_IO is "I/O possible" and by default gdb
will try and restart the guest, getting us nowhere. Report
GDB_SIGNAL_STOP instead which should at least halt the session at the
failure point.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Luis Machado <luis.machado@arm.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
---
 gdbstub/internals.h | 1 +
 gdbstub/system.c    | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/gdbstub/internals.h b/gdbstub/internals.h
index 465c24b36e..5c0c725e54 100644
--- a/gdbstub/internals.h
+++ b/gdbstub/internals.h
@@ -24,6 +24,7 @@ enum {
     GDB_SIGNAL_TRAP = 5,
     GDB_SIGNAL_ABRT = 6,
     GDB_SIGNAL_ALRM = 14,
+    GDB_SIGNAL_STOP = 17,
     GDB_SIGNAL_IO = 23,
     GDB_SIGNAL_XCPU = 24,
     GDB_SIGNAL_UNKNOWN = 143
diff --git a/gdbstub/system.c b/gdbstub/system.c
index 783ac140b9..83fd452800 100644
--- a/gdbstub/system.c
+++ b/gdbstub/system.c
@@ -183,7 +183,7 @@ static void gdb_vm_state_change(void *opaque, bool running, RunState state)
         break;
     case RUN_STATE_IO_ERROR:
         trace_gdbstub_hit_io_error();
-        ret = GDB_SIGNAL_IO;
+        ret = GDB_SIGNAL_STOP;
         break;
     case RUN_STATE_WATCHDOG:
         trace_gdbstub_hit_watchdog();
-- 
2.39.2

[PATCH v2 2/7] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS

[Alex Bennée]

It doesn't make sense to have two classes of flaky tests. While it may
take the constrained environment of CI to trigger failures easily it
doesn't mean they don't occasionally happen on developer machines. As
CI is the gating factor to passing there is no point developers
running the tests locally anyway unless they are trying to fix things.

While we are at it update the language in the docs to discourage the
QEMU_TEST_FLAKY_TESTS becoming a permanent solution.

Message-Id: <20231130153333.2424775-3-alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - be more explicit about raising bugs
  - add some bug urls
---
 docs/devel/testing.rst                   | 31 +++++++++++++++---------
 tests/avocado/boot_linux.py              |  8 +++---
 tests/avocado/boot_linux_console.py      |  5 ++--
 tests/avocado/intel_iommu.py             |  5 ++--
 tests/avocado/linux_initrd.py            |  5 ++--
 tests/avocado/machine_aspeed.py          |  8 +++---
 tests/avocado/machine_mips_malta.py      |  8 +++---
 tests/avocado/machine_rx_gdbsim.py       |  8 +++---
 tests/avocado/machine_s390_ccw_virtio.py |  2 +-
 tests/avocado/replay_kernel.py           |  6 +++--
 tests/avocado/reverse_debugging.py       | 14 +++++++----
 tests/avocado/smmu.py                    |  5 ++--
 tests/avocado/tuxrun_baselines.py        |  4 +--
 13 files changed, 68 insertions(+), 41 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 22218dbedb..76465b8f3d 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -1371,23 +1371,32 @@ conditions. For example, tests that take longer to execute when QEMU is
 compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable
 has been used to determine whether those tests should run or not.
 
-GITLAB_CI
-^^^^^^^^^
-A number of tests are flagged to not run on the GitLab CI. Usually because
-they proved to the flaky or there are constraints on the CI environment which
-would make them fail. If you encounter a similar situation then use that
-variable as shown on the code snippet below to skip the test:
+QEMU_TEST_FLAKY_TESTS
+^^^^^^^^^^^^^^^^^^^^^
+Some tests are not working reliably and thus are disabled by default.
+This includes tests that don't run reliably on GitLab's CI which
+usually expose real issues that are rarely seen on developer machines
+due to the constraints of the CI environment. If you encounter a
+similar situation then raise a bug and then mark the test as shown on
+the code snippet below:
 
 .. code::
 
-  @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+  # See https://gitlab.com/qemu-project/qemu/-/issues/nnnn
+  @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
   def test(self):
       do_something()
 
-QEMU_TEST_FLAKY_TESTS
-^^^^^^^^^^^^^^^^^^^^^
-Some tests are not working reliably and thus are disabled by default.
-Set this environment variable to enable them.
+Tests should not live in this state forever and should either be fixed
+or eventually removed.
+
+To run such tests locally you will need to set the environment
+variable. For example:
+
+.. code::
+
+   env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado run \
+      tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg
 
 Uninstalling Avocado
 ~~~~~~~~~~~~~~~~~~~~
diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py
index be30dcbd58..9e9773e6e1 100644
--- a/tests/avocado/boot_linux.py
+++ b/tests/avocado/boot_linux.py
@@ -12,7 +12,7 @@
 
 from avocado_qemu import LinuxTest, BUILD_DIR
 
-from avocado import skipIf
+from avocado import skipUnless
 
 
 class BootLinuxX8664(LinuxTest):
@@ -93,7 +93,8 @@ class BootLinuxPPC64(LinuxTest):
 
     timeout = 360
 
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_pseries_tcg(self):
         """
         :avocado: tags=machine:pseries
@@ -111,7 +112,8 @@ class BootLinuxS390X(LinuxTest):
 
     timeout = 240
 
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_s390_ccw_virtio_tcg(self):
         """
         :avocado: tags=machine:s390-ccw-virtio
diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
index 6eab515718..231b4f68e5 100644
--- a/tests/avocado/boot_linux_console.py
+++ b/tests/avocado/boot_linux_console.py
@@ -15,7 +15,7 @@
 
 from avocado import skip
 from avocado import skipUnless
-from avocado import skipIf
+from avocado import skipUnless
 from avocado_qemu import QemuSystemTest
 from avocado_qemu import exec_command
 from avocado_qemu import exec_command_and_wait_for_pattern
@@ -1419,7 +1419,8 @@ def test_ppc_mac99(self):
     # This test has a 6-10% failure rate on various hosts that look
     # like issues with a buggy kernel. As a result we don't want it
     # gating releases on Gitlab.
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_sh4_r2d(self):
         """
         :avocado: tags=arch:sh4
diff --git a/tests/avocado/intel_iommu.py b/tests/avocado/intel_iommu.py
index 77635ab56c..2dd11a6346 100644
--- a/tests/avocado/intel_iommu.py
+++ b/tests/avocado/intel_iommu.py
@@ -9,10 +9,11 @@
 # later.  See the COPYING file in the top-level directory.
 import os
 
-from avocado import skipIf
+from avocado import skipUnless
 from avocado_qemu import LinuxTest
 
-@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
 class IntelIOMMU(LinuxTest):
     """
     :avocado: tags=arch:x86_64
diff --git a/tests/avocado/linux_initrd.py b/tests/avocado/linux_initrd.py
index ba02e5a563..c40a987bd1 100644
--- a/tests/avocado/linux_initrd.py
+++ b/tests/avocado/linux_initrd.py
@@ -13,7 +13,7 @@
 import tempfile
 
 from avocado_qemu import QemuSystemTest
-from avocado import skipIf
+from avocado import skipUnless
 
 
 class LinuxInitrd(QemuSystemTest):
@@ -53,7 +53,8 @@ def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self):
                 max_size + 1)
             self.assertRegex(self.vm.get_log(), expected_msg)
 
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_with_2gib_file_should_work_with_linux_v4_16(self):
         """
         QEMU has supported up to 4 GiB initrd for recent kernel
diff --git a/tests/avocado/machine_aspeed.py b/tests/avocado/machine_aspeed.py
index df31b2a8a8..258fb50c47 100644
--- a/tests/avocado/machine_aspeed.py
+++ b/tests/avocado/machine_aspeed.py
@@ -18,7 +18,7 @@
 from avocado_qemu import interrupt_interactive_console_until_pattern
 from avocado_qemu import has_cmd
 from avocado.utils import archive
-from avocado import skipIf
+from avocado import skipUnless
 from avocado import skipUnless
 
 
@@ -311,7 +311,8 @@ def do_test_arm_aspeed_sdk_start(self, image):
             self, 'boot', '## Loading kernel from FIT Image')
         self.wait_for_console_pattern('Starting kernel ...')
 
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_arm_ast2500_evb_sdk(self):
         """
         :avocado: tags=arch:arm
@@ -329,7 +330,8 @@ def test_arm_ast2500_evb_sdk(self):
             self.workdir + '/ast2500-default/image-bmc')
         self.wait_for_console_pattern('nodistro.0 ast2500-default ttyS4')
 
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_arm_ast2600_evb_sdk(self):
         """
         :avocado: tags=arch:arm
diff --git a/tests/avocado/machine_mips_malta.py b/tests/avocado/machine_mips_malta.py
index 9bd54518bf..959dcf5602 100644
--- a/tests/avocado/machine_mips_malta.py
+++ b/tests/avocado/machine_mips_malta.py
@@ -11,7 +11,7 @@
 import gzip
 import logging
 
-from avocado import skipIf
+from avocado import skipUnless
 from avocado import skipUnless
 from avocado.utils import archive
 from avocado_qemu import QemuSystemTest
@@ -101,7 +101,8 @@ def test_mips_malta_i6400_framebuffer_logo_1core(self):
         """
         self.do_test_i6400_framebuffer_logo(1)
 
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_mips_malta_i6400_framebuffer_logo_7cores(self):
         """
         :avocado: tags=arch:mips64el
@@ -111,7 +112,8 @@ def test_mips_malta_i6400_framebuffer_logo_7cores(self):
         """
         self.do_test_i6400_framebuffer_logo(7)
 
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_mips_malta_i6400_framebuffer_logo_8cores(self):
         """
         :avocado: tags=arch:mips64el
diff --git a/tests/avocado/machine_rx_gdbsim.py b/tests/avocado/machine_rx_gdbsim.py
index 6cd8704b01..350a73fbbb 100644
--- a/tests/avocado/machine_rx_gdbsim.py
+++ b/tests/avocado/machine_rx_gdbsim.py
@@ -10,7 +10,7 @@
 
 import os
 
-from avocado import skipIf
+from avocado import skipUnless
 from avocado_qemu import QemuSystemTest
 from avocado_qemu import exec_command_and_wait_for_pattern
 from avocado_qemu import wait_for_console_pattern
@@ -22,7 +22,8 @@ class RxGdbSimMachine(QemuSystemTest):
     timeout = 30
     KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
 
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_uboot(self):
         """
         U-Boot and checks that the console is operational.
@@ -46,7 +47,8 @@ def test_uboot(self):
         # FIXME limit baudrate on chardev, else we type too fast
         #exec_command_and_wait_for_pattern(self, 'version', gcc_version)
 
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_linux_sash(self):
         """
         Boots a Linux kernel and checks that the console is operational.
diff --git a/tests/avocado/machine_s390_ccw_virtio.py b/tests/avocado/machine_s390_ccw_virtio.py
index ffd914ded9..61e75d8f9e 100644
--- a/tests/avocado/machine_s390_ccw_virtio.py
+++ b/tests/avocado/machine_s390_ccw_virtio.py
@@ -12,7 +12,7 @@
 import os
 import tempfile
 
-from avocado import skipIf
+from avocado import skipUnless
 from avocado_qemu import QemuSystemTest
 from avocado_qemu import exec_command_and_wait_for_pattern
 from avocado_qemu import wait_for_console_pattern
diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py
index 53cb7e5091..0d32cc280e 100644
--- a/tests/avocado/replay_kernel.py
+++ b/tests/avocado/replay_kernel.py
@@ -15,7 +15,7 @@
 import time
 
 from avocado import skip
-from avocado import skipIf
+from avocado import skipUnless
 from avocado import skipUnless
 from avocado_qemu import wait_for_console_pattern
 from avocado.utils import archive
@@ -82,6 +82,7 @@ def run_rr(self, kernel_path, kernel_command_line, console_pattern,
 
 class ReplayKernelNormal(ReplayKernelBase):
 
+    # See https://gitlab.com/qemu-project/qemu/-/issues/2010
     @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test sometimes gets stuck')
     def test_x86_64_pc(self):
         """
@@ -179,7 +180,8 @@ def test_arm_virt(self):
 
         self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1)
 
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_arm_cubieboard_initrd(self):
         """
         :avocado: tags=arch:arm
diff --git a/tests/avocado/reverse_debugging.py b/tests/avocado/reverse_debugging.py
index ed04e92bb4..9a468321e5 100644
--- a/tests/avocado/reverse_debugging.py
+++ b/tests/avocado/reverse_debugging.py
@@ -10,7 +10,7 @@
 import os
 import logging
 
-from avocado import skipIf
+from avocado import skipUnless
 from avocado_qemu import BUILD_DIR
 from avocado.utils import datadrainer
 from avocado.utils import gdb
@@ -206,7 +206,8 @@ def get_pc(self, g):
             + self.get_reg_le(g, self.REG_CS) * 0x10
 
     # unidentified gitlab timeout problem
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_x86_64_pc(self):
         """
         :avocado: tags=arch:x86_64
@@ -223,7 +224,8 @@ class ReverseDebugging_AArch64(ReverseDebugging):
     REG_PC = 32
 
     # unidentified gitlab timeout problem
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_aarch64_virt(self):
         """
         :avocado: tags=arch:aarch64
@@ -247,7 +249,8 @@ class ReverseDebugging_ppc64(ReverseDebugging):
     REG_PC = 0x40
 
     # unidentified gitlab timeout problem
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_ppc64_pseries(self):
         """
         :avocado: tags=arch:ppc64
@@ -260,7 +263,8 @@ def test_ppc64_pseries(self):
         self.reverse_debugging()
 
     # See https://gitlab.com/qemu-project/qemu/-/issues/1992
-    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
     def test_ppc64_powernv(self):
         """
         :avocado: tags=arch:ppc64
diff --git a/tests/avocado/smmu.py b/tests/avocado/smmu.py
index b3c4de6bf4..05b34418a5 100644
--- a/tests/avocado/smmu.py
+++ b/tests/avocado/smmu.py
@@ -9,10 +9,11 @@
 # later.  See the COPYING file in the top-level directory.
 import os
 
-from avocado import skipIf
+from avocado import skipUnless
 from avocado_qemu import LinuxTest, BUILD_DIR
 
-@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
 class SMMU(LinuxTest):
     """
     :avocado: tags=accel:kvm
diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py
index c99bea6c0b..5f859f4e6f 100644
--- a/tests/avocado/tuxrun_baselines.py
+++ b/tests/avocado/tuxrun_baselines.py
@@ -13,7 +13,7 @@
 import time
 import tempfile
 
-from avocado import skip, skipIf
+from avocado import skip, skipUnless
 from avocado_qemu import QemuSystemTest
 from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
 from avocado_qemu import wait_for_console_pattern
@@ -551,7 +551,7 @@ def test_s390(self):
                            haltmsg="Requesting system halt")
 
     # Note: some segfaults caused by unaligned userspace access
-    @skipIf(os.getenv('GITLAB_CI'), 'Skipping unstable test on GitLab')
+    @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
     def test_sh4(self):
         """
         :avocado: tags=arch:sh4
-- 
2.39.2

[PATCH v2 3/7] tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test

[Alex Bennée]

The assets are no longer archived by Debian so we can't run this on
CI. While some people may still have the test in their cache we do
have more recent images from tuxrun so this isn't a great loss.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 tests/avocado/boot_linux_console.py | 22 ----------------------
 1 file changed, 22 deletions(-)

diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
index 231b4f68e5..5d978f6dd0 100644
--- a/tests/avocado/boot_linux_console.py
+++ b/tests/avocado/boot_linux_console.py
@@ -116,28 +116,6 @@ def test_x86_64_pc(self):
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
 
-    def test_mips_malta(self):
-        """
-        :avocado: tags=arch:mips
-        :avocado: tags=machine:malta
-        :avocado: tags=endian:big
-        """
-        deb_url = ('http://snapshot.debian.org/archive/debian/'
-                   '20130217T032700Z/pool/main/l/linux-2.6/'
-                   'linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb')
-        deb_hash = 'a8cfc28ad8f45f54811fc6cf74fc43ffcfe0ba04'
-        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
-        kernel_path = self.extract_from_deb(deb_path,
-                                            '/boot/vmlinux-2.6.32-5-4kc-malta')
-
-        self.vm.set_console()
-        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE + 'console=ttyS0'
-        self.vm.add_args('-kernel', kernel_path,
-                         '-append', kernel_command_line)
-        self.vm.launch()
-        console_pattern = 'Kernel command line: %s' % kernel_command_line
-        self.wait_for_console_pattern(console_pattern)
-
     def test_mips64el_malta(self):
         """
         This test requires the ar tool to extract "data.tar.gz" from
-- 
2.39.2

Re: [PATCH v2 3/7] tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2213 bytes --]

On Fri, Dec 1, 2023, 04:38 Alex Bennée <alex.bennee@linaro.org> wrote:

> The assets are no longer archived by Debian so we can't run this on
> CI. While some people may still have the test in their cache we do
> have more recent images from tuxrun so this isn't a great loss.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Stefan Hajnoczi <stefanha@redhat.com>
> Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>  tests/avocado/boot_linux_console.py | 22 ----------------------
>  1 file changed, 22 deletions(-)
>

Acked-by: Stefan Hajnoczi <stefanha@redhat.com>


> diff --git a/tests/avocado/boot_linux_console.py
> b/tests/avocado/boot_linux_console.py
> index 231b4f68e5..5d978f6dd0 100644
> --- a/tests/avocado/boot_linux_console.py
> +++ b/tests/avocado/boot_linux_console.py
> @@ -116,28 +116,6 @@ def test_x86_64_pc(self):
>          console_pattern = 'Kernel command line: %s' % kernel_command_line
>          self.wait_for_console_pattern(console_pattern)
>
> -    def test_mips_malta(self):
> -        """
> -        :avocado: tags=arch:mips
> -        :avocado: tags=machine:malta
> -        :avocado: tags=endian:big
> -        """
> -        deb_url = ('http://snapshot.debian.org/archive/debian/'
> -                   '20130217T032700Z/pool/main/l/linux-2.6/'
> -                   'linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb')
> -        deb_hash = 'a8cfc28ad8f45f54811fc6cf74fc43ffcfe0ba04'
> -        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
> -        kernel_path = self.extract_from_deb(deb_path,
> -
> '/boot/vmlinux-2.6.32-5-4kc-malta')
> -
> -        self.vm.set_console()
> -        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE +
> 'console=ttyS0'
> -        self.vm.add_args('-kernel', kernel_path,
> -                         '-append', kernel_command_line)
> -        self.vm.launch()
> -        console_pattern = 'Kernel command line: %s' % kernel_command_line
> -        self.wait_for_console_pattern(console_pattern)
> -
>      def test_mips64el_malta(self):
>          """
>          This test requires the ar tool to extract "data.tar.gz" from
> --
> 2.39.2
>
>
>

[-- Attachment #2: Type: text/html, Size: 3533 bytes --]

Re: [PATCH v2 3/7] tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test

[Philippe Mathieu-Daudé]

On 1/12/23 10:36, Alex Bennée wrote:
> The assets are no longer archived by Debian so we can't run this on
> CI. While some people may still have the test in their cache we do
> have more recent images from tuxrun so this isn't a great loss.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Stefan Hajnoczi <stefanha@redhat.com>
> Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   tests/avocado/boot_linux_console.py | 22 ----------------------
>   1 file changed, 22 deletions(-)
> 
> diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
> index 231b4f68e5..5d978f6dd0 100644
> --- a/tests/avocado/boot_linux_console.py
> +++ b/tests/avocado/boot_linux_console.py
> @@ -116,28 +116,6 @@ def test_x86_64_pc(self):
>           console_pattern = 'Kernel command line: %s' % kernel_command_line
>           self.wait_for_console_pattern(console_pattern)
>   
> -    def test_mips_malta(self):
> -        """
> -        :avocado: tags=arch:mips
> -        :avocado: tags=machine:malta
> -        :avocado: tags=endian:big
> -        """
> -        deb_url = ('http://snapshot.debian.org/archive/debian/'
> -                   '20130217T032700Z/pool/main/l/linux-2.6/'
> -                   'linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb')

You are dropping test_mips_malta() while the subject says
"test_mips_malta_cpio" which uses:

         deb_url = ('http://snapshot.debian.org/archive/debian/'
                    '20160601T041800Z/pool/main/l/linux/'
                    'linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb')

Anyway, the lesson is we can't rely on Debian snapshot mirror
anymore, so more need to be cleaned:

$ git grep url.*snapshot.debian.org
tests/avocado/boot_linux_console.py:127:        deb_url = 
('http://snapshot.debian.org/archive/debian/'
tests/avocado/boot_linux_console.py:160:        deb_url = 
('http://snapshot.debian.org/archive/debian/'
tests/avocado/boot_linux_console.py:203:        deb_url = 
('http://snapshot.debian.org/archive/debian/'
tests/avocado/boot_linux_console.py:513:        deb_url = 
('https://snapshot.debian.org/archive/debian/'
tests/avocado/boot_linux_console.py:1129:        deb_url = 
('http://snapshot.debian.org/archive/debian/'
tests/avocado/boot_linux_console.py:1255:        deb_url = 
('https://snapshot.debian.org/archive/debian-ports'
tests/avocado/boot_linux_console.py:1299:        deb_url = 
('http://snapshot.debian.org/archive/debian/'
tests/avocado/machine_s390_ccw_virtio.py:50:        kernel_url = 
('https://snapshot.debian.org/archive/debian/'
tests/avocado/machine_s390_ccw_virtio.py:56:        initrd_url = 
('https://snapshot.debian.org/archive/debian/'
tests/avocado/replay_kernel.py:110:        deb_url = 
('http://snapshot.debian.org/archive/debian/'
tests/avocado/replay_kernel.py:139:        deb_url = 
('http://snapshot.debian.org/archive/debian/'
tests/avocado/replay_kernel.py:287:        deb_url = 
('https://snapshot.debian.org/archive/debian-ports'
tests/avocado/replay_kernel.py:445:        deb_url = 
('http://snapshot.debian.org/archive/debian/'

Re: [PATCH v2 3/7] tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test

[Philippe Mathieu-Daudé]

On 1/12/23 14:45, Philippe Mathieu-Daudé wrote:
> On 1/12/23 10:36, Alex Bennée wrote:
>> The assets are no longer archived by Debian so we can't run this on
>> CI. While some people may still have the test in their cache we do
>> have more recent images from tuxrun so this isn't a great loss.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Cc: Stefan Hajnoczi <stefanha@redhat.com>
>> Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
>> ---
>>   tests/avocado/boot_linux_console.py | 22 ----------------------
>>   1 file changed, 22 deletions(-)
>>
>> diff --git a/tests/avocado/boot_linux_console.py 
>> b/tests/avocado/boot_linux_console.py
>> index 231b4f68e5..5d978f6dd0 100644
>> --- a/tests/avocado/boot_linux_console.py
>> +++ b/tests/avocado/boot_linux_console.py
>> @@ -116,28 +116,6 @@ def test_x86_64_pc(self):
>>           console_pattern = 'Kernel command line: %s' % 
>> kernel_command_line
>>           self.wait_for_console_pattern(console_pattern)
>> -    def test_mips_malta(self):
>> -        """
>> -        :avocado: tags=arch:mips
>> -        :avocado: tags=machine:malta
>> -        :avocado: tags=endian:big
>> -        """
>> -        deb_url = ('http://snapshot.debian.org/archive/debian/'
>> -                   '20130217T032700Z/pool/main/l/linux-2.6/'
>> -                   'linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb')

This link still works for me, what am I missing?

$ wget 
http://snapshot.debian.org/archive/debian/20130217T032700Z/pool/main/l/linux-2.6/linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb
--2023-12-01 14:55:31-- 
http://snapshot.debian.org/archive/debian/20130217T032700Z/pool/main/l/linux-2.6/linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb
Resolving snapshot.debian.org (snapshot.debian.org)... 185.17.185.185, 
193.62.202.27
Connecting to snapshot.debian.org 
(snapshot.debian.org)|185.17.185.185|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30231730 (29M)
Saving to: ‘linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb’

    linux-image-2.6.32-5-4kc-ma  12%[=>    ]   3,71M  1,23MB/s    eta 20s

> 
> You are dropping test_mips_malta() while the subject says
> "test_mips_malta_cpio" which uses:
> 
>          deb_url = ('http://snapshot.debian.org/archive/debian/'
>                     '20160601T041800Z/pool/main/l/linux/'
>                     'linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb')
> 
> Anyway, the lesson is we can't rely on Debian snapshot mirror
> anymore, so more need to be cleaned:
> 
> $ git grep url.*snapshot.debian.org
> tests/avocado/boot_linux_console.py:127:        deb_url = 
> ('http://snapshot.debian.org/archive/debian/'
> tests/avocado/boot_linux_console.py:160:        deb_url = 
> ('http://snapshot.debian.org/archive/debian/'
> tests/avocado/boot_linux_console.py:203:        deb_url = 
> ('http://snapshot.debian.org/archive/debian/'
> tests/avocado/boot_linux_console.py:513:        deb_url = 
> ('https://snapshot.debian.org/archive/debian/'
> tests/avocado/boot_linux_console.py:1129:        deb_url = 
> ('http://snapshot.debian.org/archive/debian/'
> tests/avocado/boot_linux_console.py:1255:        deb_url = 
> ('https://snapshot.debian.org/archive/debian-ports'
> tests/avocado/boot_linux_console.py:1299:        deb_url = 
> ('http://snapshot.debian.org/archive/debian/'
> tests/avocado/machine_s390_ccw_virtio.py:50:        kernel_url = 
> ('https://snapshot.debian.org/archive/debian/'
> tests/avocado/machine_s390_ccw_virtio.py:56:        initrd_url = 
> ('https://snapshot.debian.org/archive/debian/'
> tests/avocado/replay_kernel.py:110:        deb_url = 
> ('http://snapshot.debian.org/archive/debian/'
> tests/avocado/replay_kernel.py:139:        deb_url = 
> ('http://snapshot.debian.org/archive/debian/'
> tests/avocado/replay_kernel.py:287:        deb_url = 
> ('https://snapshot.debian.org/archive/debian-ports'
> tests/avocado/replay_kernel.py:445:        deb_url = 
> ('http://snapshot.debian.org/archive/debian/'

Re: [PATCH v2 3/7] tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test

[Philippe Mathieu-Daudé]

On 1/12/23 14:57, Philippe Mathieu-Daudé wrote:
> On 1/12/23 14:45, Philippe Mathieu-Daudé wrote:
>> On 1/12/23 10:36, Alex Bennée wrote:
>>> The assets are no longer archived by Debian so we can't run this on
>>> CI. While some people may still have the test in their cache we do
>>> have more recent images from tuxrun so this isn't a great loss.
>>>
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> Cc: Stefan Hajnoczi <stefanha@redhat.com>
>>> Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
>>> ---
>>>   tests/avocado/boot_linux_console.py | 22 ----------------------
>>>   1 file changed, 22 deletions(-)
>>>
>>> diff --git a/tests/avocado/boot_linux_console.py 
>>> b/tests/avocado/boot_linux_console.py
>>> index 231b4f68e5..5d978f6dd0 100644
>>> --- a/tests/avocado/boot_linux_console.py
>>> +++ b/tests/avocado/boot_linux_console.py
>>> @@ -116,28 +116,6 @@ def test_x86_64_pc(self):
>>>           console_pattern = 'Kernel command line: %s' % 
>>> kernel_command_line
>>>           self.wait_for_console_pattern(console_pattern)
>>> -    def test_mips_malta(self):
>>> -        """
>>> -        :avocado: tags=arch:mips
>>> -        :avocado: tags=machine:malta
>>> -        :avocado: tags=endian:big
>>> -        """
>>> -        deb_url = ('http://snapshot.debian.org/archive/debian/'
>>> -                   '20130217T032700Z/pool/main/l/linux-2.6/'
>>> -                   'linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb')
> 
> This link still works for me, what am I missing?
> 
> $ wget 
> http://snapshot.debian.org/archive/debian/20130217T032700Z/pool/main/l/linux-2.6/linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb
> --2023-12-01 14:55:31-- 
> http://snapshot.debian.org/archive/debian/20130217T032700Z/pool/main/l/linux-2.6/linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb
> Resolving snapshot.debian.org (snapshot.debian.org)... 185.17.185.185, 
> 193.62.202.27
> Connecting to snapshot.debian.org 
> (snapshot.debian.org)|185.17.185.185|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 30231730 (29M)
> Saving to: ‘linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb’
> 
>     linux-image-2.6.32-5-4kc-ma  12%[=>    ]   3,71M  1,23MB/s    eta 20s

Sharing HTTP headers:

$ curl -v 
http://snapshot.debian.org/archive/debian/20130217T032700Z/pool/main/l/linux-2.6/linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb
*   Trying 193.62.202.27:80...
* Connected to snapshot.debian.org (193.62.202.27) port 80 (#0)
 > GET 
/archive/debian/20130217T032700Z/pool/main/l/linux-2.6/linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb 
HTTP/1.1
 > Host: snapshot.debian.org
 > User-Agent: curl/8.1.2
 > Accept: */*
 >
< HTTP/1.1 200 OK
< Date: Fri, 01 Dec 2023 13:54:55 GMT
< Server: Apache
< X-Content-Type-Options: nosniff
< X-Frame-Options: sameorigin
< Referrer-Policy: no-referrer
< X-Xss-Protection: 1
< Permissions-Policy: interest-cohort=()
< Last-Modified: Sun, 17 Feb 2013 03:34:37 GMT
< ETag: "1cd4cb2-4d5e349b3403e"
< Accept-Ranges: bytes
< Content-Length: 30231730
< X-Clacks-Overhead: GNU Terry Pratchett
< Cache-Control: max-age=31536000, public
< X-Varnish: 383440315
< Age: 0
< Via: 1.1 varnish (Varnish/6.5)
< connection: close

>> Anyway, the lesson is we can't rely on Debian snapshot mirror
>> anymore, so more need to be cleaned:
>>
>> $ git grep url.*snapshot.debian.org
>> tests/avocado/boot_linux_console.py:127:        deb_url = 
>> ('http://snapshot.debian.org/archive/debian/'
>> tests/avocado/boot_linux_console.py:160:        deb_url = 
>> ('http://snapshot.debian.org/archive/debian/'
>> tests/avocado/boot_linux_console.py:203:        deb_url = 
>> ('http://snapshot.debian.org/archive/debian/'
>> tests/avocado/boot_linux_console.py:513:        deb_url = 
>> ('https://snapshot.debian.org/archive/debian/'
>> tests/avocado/boot_linux_console.py:1129:        deb_url = 
>> ('http://snapshot.debian.org/archive/debian/'
>> tests/avocado/boot_linux_console.py:1255:        deb_url = 
>> ('https://snapshot.debian.org/archive/debian-ports'
>> tests/avocado/boot_linux_console.py:1299:        deb_url = 
>> ('http://snapshot.debian.org/archive/debian/'
>> tests/avocado/machine_s390_ccw_virtio.py:50:        kernel_url = 
>> ('https://snapshot.debian.org/archive/debian/'
>> tests/avocado/machine_s390_ccw_virtio.py:56:        initrd_url = 
>> ('https://snapshot.debian.org/archive/debian/'
>> tests/avocado/replay_kernel.py:110:        deb_url = 
>> ('http://snapshot.debian.org/archive/debian/'
>> tests/avocado/replay_kernel.py:139:        deb_url = 
>> ('http://snapshot.debian.org/archive/debian/'
>> tests/avocado/replay_kernel.py:287:        deb_url = 
>> ('https://snapshot.debian.org/archive/debian-ports'
>> tests/avocado/replay_kernel.py:445:        deb_url = 
>> ('http://snapshot.debian.org/archive/debian/'
> 

Re: [PATCH v2 3/7] tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@linaro.org> writes:

> On 1/12/23 14:45, Philippe Mathieu-Daudé wrote:
>> On 1/12/23 10:36, Alex Bennée wrote:
>>> The assets are no longer archived by Debian so we can't run this on
>>> CI. While some people may still have the test in their cache we do
>>> have more recent images from tuxrun so this isn't a great loss.
>>>
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> Cc: Stefan Hajnoczi <stefanha@redhat.com>
>>> Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
>>> ---
>>>   tests/avocado/boot_linux_console.py | 22 ----------------------
>>>   1 file changed, 22 deletions(-)
>>>
>>> diff --git a/tests/avocado/boot_linux_console.py
>>> b/tests/avocado/boot_linux_console.py
>>> index 231b4f68e5..5d978f6dd0 100644
>>> --- a/tests/avocado/boot_linux_console.py
>>> +++ b/tests/avocado/boot_linux_console.py
>>> @@ -116,28 +116,6 @@ def test_x86_64_pc(self):
>>>           console_pattern = 'Kernel command line: %s' %
>>> kernel_command_line
>>>           self.wait_for_console_pattern(console_pattern)
>>> -    def test_mips_malta(self):
>>> -        """
>>> -        :avocado: tags=arch:mips
>>> -        :avocado: tags=machine:malta
>>> -        :avocado: tags=endian:big
>>> -        """
>>> -        deb_url = ('http://snapshot.debian.org/archive/debian/'
>>> -                   '20130217T032700Z/pool/main/l/linux-2.6/'
>>> -                   'linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb')
>
> This link still works for me, what am I missing?
>
> $ wget
> http://snapshot.debian.org/archive/debian/20130217T032700Z/pool/main/l/linux-2.6/linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb
> --2023-12-01 14:55:31--
>   http://snapshot.debian.org/archive/debian/20130217T032700Z/pool/main/l/linux-2.6/linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb
> Resolving snapshot.debian.org (snapshot.debian.org)... 185.17.185.185,
> 193.62.202.27
> Connecting to snapshot.debian.org
> (snapshot.debian.org)|185.17.185.185|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 30231730 (29M)
> Saving to: ‘linux-image-2.6.32-5-4kc-malta_2.6.32-48_mips.deb’

The failing link is the malta_cpio linux-4.7 one, I've fixed that in the
PR.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

[PATCH v2 4/7] tests/avocado: drop malta yamon tests

[Alex Bennée]

The assets are no longer available on the website so these are
blocking CI.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 tests/avocado/machine_mips_malta.py | 37 -----------------------------
 1 file changed, 37 deletions(-)

diff --git a/tests/avocado/machine_mips_malta.py b/tests/avocado/machine_mips_malta.py
index 959dcf5602..3a1ec85c20 100644
--- a/tests/avocado/machine_mips_malta.py
+++ b/tests/avocado/machine_mips_malta.py
@@ -122,40 +122,3 @@ def test_mips_malta_i6400_framebuffer_logo_8cores(self):
         :avocado: tags=mips:smp
         """
         self.do_test_i6400_framebuffer_logo(8)
-
-class MaltaMachine(QemuSystemTest):
-
-    def do_test_yamon(self):
-        rom_url = ('http://www.imgtec.com/tools/mips-tools/downloads/'
-                   'yamon/yamon-bin-02.22.zip')
-        rom_hash = '8da7ecddbc5312704b8b324341ee238189bde480'
-        zip_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
-
-        archive.extract(zip_path, self.workdir)
-        yamon_path = os.path.join(self.workdir, 'yamon-02.22.bin')
-
-        self.vm.set_console()
-        self.vm.add_args('-bios', yamon_path)
-        self.vm.launch()
-
-        prompt =  'YAMON>'
-        pattern = 'YAMON ROM Monitor'
-        interrupt_interactive_console_until_pattern(self, pattern, prompt)
-        wait_for_console_pattern(self, prompt)
-        self.vm.shutdown()
-
-    def test_mipsel_malta_yamon(self):
-        """
-        :avocado: tags=arch:mipsel
-        :avocado: tags=machine:malta
-        :avocado: tags=endian:little
-        """
-        self.do_test_yamon()
-
-    def test_mips64el_malta_yamon(self):
-        """
-        :avocado: tags=arch:mips64el
-        :avocado: tags=machine:malta
-        :avocado: tags=endian:little
-        """
-        self.do_test_yamon()
-- 
2.39.2

Re: [PATCH v2 4/7] tests/avocado: drop malta yamon tests

[Philippe Mathieu-Daudé]

Hi Alex,

On 1/12/23 10:36, Alex Bennée wrote:
> The assets are no longer available on the website so these are
> blocking CI.
> 

How are these "blocking CI"? Missing artifact wasn't be fatal,
is it now? Also, did the artifact cache got flushed? These tests
pass locally, I disagree with removing them.

I can send a patch using YAMON_PATH like we have with RESCUE_YL_PATH,
but I still consider missing artifact shouldn't be an issue. We are
missing the point of the Avocado cache and the possibility to manually
add artifacts.

> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>   tests/avocado/machine_mips_malta.py | 37 -----------------------------
>   1 file changed, 37 deletions(-)
> 
> diff --git a/tests/avocado/machine_mips_malta.py b/tests/avocado/machine_mips_malta.py
> index 959dcf5602..3a1ec85c20 100644
> --- a/tests/avocado/machine_mips_malta.py
> +++ b/tests/avocado/machine_mips_malta.py
> @@ -122,40 +122,3 @@ def test_mips_malta_i6400_framebuffer_logo_8cores(self):
>           :avocado: tags=mips:smp
>           """
>           self.do_test_i6400_framebuffer_logo(8)
> -
> -class MaltaMachine(QemuSystemTest):
> -
> -    def do_test_yamon(self):
> -        rom_url = ('http://www.imgtec.com/tools/mips-tools/downloads/'
> -                   'yamon/yamon-bin-02.22.zip')
> -        rom_hash = '8da7ecddbc5312704b8b324341ee238189bde480'
> -        zip_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
> -
> -        archive.extract(zip_path, self.workdir)
> -        yamon_path = os.path.join(self.workdir, 'yamon-02.22.bin')
> -
> -        self.vm.set_console()
> -        self.vm.add_args('-bios', yamon_path)
> -        self.vm.launch()
> -
> -        prompt =  'YAMON>'
> -        pattern = 'YAMON ROM Monitor'
> -        interrupt_interactive_console_until_pattern(self, pattern, prompt)
> -        wait_for_console_pattern(self, prompt)
> -        self.vm.shutdown()
> -
> -    def test_mipsel_malta_yamon(self):
> -        """
> -        :avocado: tags=arch:mipsel
> -        :avocado: tags=machine:malta
> -        :avocado: tags=endian:little
> -        """
> -        self.do_test_yamon()
> -
> -    def test_mips64el_malta_yamon(self):
> -        """
> -        :avocado: tags=arch:mips64el
> -        :avocado: tags=machine:malta
> -        :avocado: tags=endian:little
> -        """
> -        self.do_test_yamon()

Re: [PATCH v2 4/7] tests/avocado: drop malta yamon tests

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@linaro.org> writes:

> Hi Alex,
>
> On 1/12/23 10:36, Alex Bennée wrote:
>> The assets are no longer available on the website so these are
>> blocking CI.
>> 
>
> How are these "blocking CI"? Missing artifact wasn't be fatal,
> is it now? Also, did the artifact cache got flushed? These tests
> pass locally, I disagree with removing them.
>
> I can send a patch using YAMON_PATH like we have with RESCUE_YL_PATH,
> but I still consider missing artifact shouldn't be an issue. We are
> missing the point of the Avocado cache and the possibility to manually
> add artifacts.

Sure but in this case the binaries are gone, you can't share them and no
one else can ever run the test. At that point it just becomes dead
weight in the repository. This doesn't stop you keeping your own branch
where old tests live on while close to a warm cache but it does somewhat
limit the use of the test to the wider community.

>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
>> ---
>>   tests/avocado/machine_mips_malta.py | 37 -----------------------------
>>   1 file changed, 37 deletions(-)
>> diff --git a/tests/avocado/machine_mips_malta.py
>> b/tests/avocado/machine_mips_malta.py
>> index 959dcf5602..3a1ec85c20 100644
>> --- a/tests/avocado/machine_mips_malta.py
>> +++ b/tests/avocado/machine_mips_malta.py
>> @@ -122,40 +122,3 @@ def test_mips_malta_i6400_framebuffer_logo_8cores(self):
>>           :avocado: tags=mips:smp
>>           """
>>           self.do_test_i6400_framebuffer_logo(8)
>> -
>> -class MaltaMachine(QemuSystemTest):
>> -
>> -    def do_test_yamon(self):
>> -        rom_url = ('http://www.imgtec.com/tools/mips-tools/downloads/'
>> -                   'yamon/yamon-bin-02.22.zip')
>> -        rom_hash = '8da7ecddbc5312704b8b324341ee238189bde480'
>> -        zip_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
>> -
>> -        archive.extract(zip_path, self.workdir)
>> -        yamon_path = os.path.join(self.workdir, 'yamon-02.22.bin')
>> -
>> -        self.vm.set_console()
>> -        self.vm.add_args('-bios', yamon_path)
>> -        self.vm.launch()
>> -
>> -        prompt =  'YAMON>'
>> -        pattern = 'YAMON ROM Monitor'
>> -        interrupt_interactive_console_until_pattern(self, pattern, prompt)
>> -        wait_for_console_pattern(self, prompt)
>> -        self.vm.shutdown()
>> -
>> -    def test_mipsel_malta_yamon(self):
>> -        """
>> -        :avocado: tags=arch:mipsel
>> -        :avocado: tags=machine:malta
>> -        :avocado: tags=endian:little
>> -        """
>> -        self.do_test_yamon()
>> -
>> -    def test_mips64el_malta_yamon(self):
>> -        """
>> -        :avocado: tags=arch:mips64el
>> -        :avocado: tags=machine:malta
>> -        :avocado: tags=endian:little
>> -        """
>> -        self.do_test_yamon()

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v2 4/7] tests/avocado: drop malta yamon tests

[Philippe Mathieu-Daudé]

Hi Alex,

On 1/12/23 13:49, Alex Bennée wrote:
> Philippe Mathieu-Daudé <philmd@linaro.org> writes:
> 
>> Hi Alex,
>>
>> On 1/12/23 10:36, Alex Bennée wrote:
>>> The assets are no longer available on the website so these are
>>> blocking CI.
>>>
>>
>> How are these "blocking CI"? Missing artifact wasn't be fatal,
>> is it now? Also, did the artifact cache got flushed? These tests
>> pass locally, I disagree with removing them.
>>
>> I can send a patch using YAMON_PATH like we have with RESCUE_YL_PATH,
>> but I still consider missing artifact shouldn't be an issue. We are
>> missing the point of the Avocado cache and the possibility to manually
>> add artifacts.
> 
> Sure but in this case the binaries are gone, you can't share them and no
> one else can ever run the test. At that point it just becomes dead
> weight in the repository. This doesn't stop you keeping your own branch
> where old tests live on while close to a warm cache but it does somewhat
> limit the use of the test to the wider community.

No, I still disagree. The tests/avocado/ directory started as a place
to share tests, not to add CI gating tests. That was discussed again
2 or 3 years ago, we even recommended to change from the "opt-out on CI"
policy to the "opt-in for Gating CI". Daniel suggested the Tiers
approach, having only Tier-1 gating.

I think the mistake is to consider all tests as gating.

Why don't you want to share QEMU tests in the QEMU repository?

We might have some misunderstanding on what tests/ is for, so let's
discuss...

Regards,

Phil.

Re: [PATCH v2 4/7] tests/avocado: drop malta yamon tests

[Philippe Mathieu-Daudé]

On 1/12/23 13:49, Alex Bennée wrote:
> Philippe Mathieu-Daudé <philmd@linaro.org> writes:
> 
>> Hi Alex,
>>
>> On 1/12/23 10:36, Alex Bennée wrote:
>>> The assets are no longer available on the website so these are
>>> blocking CI.
>>>
>>
>> How are these "blocking CI"? Missing artifact wasn't be fatal,
>> is it now? Also, did the artifact cache got flushed? These tests
>> pass locally, I disagree with removing them.
>>
>> I can send a patch using YAMON_PATH like we have with RESCUE_YL_PATH,
>> but I still consider missing artifact shouldn't be an issue. We are
>> missing the point of the Avocado cache and the possibility to manually
>> add artifacts.
> 
> Sure but in this case the binaries are gone, you can't share them and no

Per https://mips.com/develop/tools/boot-loaders/:
"To support existing users of these, and the QEMU project, YAMON is now 
available under the GPL License."

Their link https://mips.com/downloads/yamon-version-02-22/ is incorrect,
Mark emailed the webmaster to ask for the correct one.

Mark, any update?

> one else can ever run the test. At that point it just becomes dead
> weight in the repository. This doesn't stop you keeping your own branch
> where old tests live on while close to a warm cache but it does somewhat
> limit the use of the test to the wider community.
> 
>>
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
>>> ---
>>>    tests/avocado/machine_mips_malta.py | 37 -----------------------------
>>>    1 file changed, 37 deletions(-)
>>> diff --git a/tests/avocado/machine_mips_malta.py
>>> b/tests/avocado/machine_mips_malta.py
>>> index 959dcf5602..3a1ec85c20 100644
>>> --- a/tests/avocado/machine_mips_malta.py
>>> +++ b/tests/avocado/machine_mips_malta.py
>>> @@ -122,40 +122,3 @@ def test_mips_malta_i6400_framebuffer_logo_8cores(self):
>>>            :avocado: tags=mips:smp
>>>            """
>>>            self.do_test_i6400_framebuffer_logo(8)
>>> -
>>> -class MaltaMachine(QemuSystemTest):
>>> -
>>> -    def do_test_yamon(self):
>>> -        rom_url = ('http://www.imgtec.com/tools/mips-tools/downloads/'
>>> -                   'yamon/yamon-bin-02.22.zip')
>>> -        rom_hash = '8da7ecddbc5312704b8b324341ee238189bde480'
>>> -        zip_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
>>> -
>>> -        archive.extract(zip_path, self.workdir)
>>> -        yamon_path = os.path.join(self.workdir, 'yamon-02.22.bin')
>>> -
>>> -        self.vm.set_console()
>>> -        self.vm.add_args('-bios', yamon_path)
>>> -        self.vm.launch()
>>> -
>>> -        prompt =  'YAMON>'
>>> -        pattern = 'YAMON ROM Monitor'
>>> -        interrupt_interactive_console_until_pattern(self, pattern, prompt)
>>> -        wait_for_console_pattern(self, prompt)
>>> -        self.vm.shutdown()
>>> -
>>> -    def test_mipsel_malta_yamon(self):
>>> -        """
>>> -        :avocado: tags=arch:mipsel
>>> -        :avocado: tags=machine:malta
>>> -        :avocado: tags=endian:little
>>> -        """
>>> -        self.do_test_yamon()
>>> -
>>> -    def test_mips64el_malta_yamon(self):
>>> -        """
>>> -        :avocado: tags=arch:mips64el
>>> -        :avocado: tags=machine:malta
>>> -        :avocado: tags=endian:little
>>> -        """
>>> -        self.do_test_yamon()
> 

Re: [PATCH v2 4/7] tests/avocado: drop malta yamon tests

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@linaro.org> writes:

> On 1/12/23 13:49, Alex Bennée wrote:
>> Philippe Mathieu-Daudé <philmd@linaro.org> writes:
>> 
>>> Hi Alex,
>>>
>>> On 1/12/23 10:36, Alex Bennée wrote:
>>>> The assets are no longer available on the website so these are
>>>> blocking CI.
>>>>
>>>
>>> How are these "blocking CI"? Missing artifact wasn't be fatal,
>>> is it now? Also, did the artifact cache got flushed? These tests
>>> pass locally, I disagree with removing them.
>>>
>>> I can send a patch using YAMON_PATH like we have with RESCUE_YL_PATH,
>>> but I still consider missing artifact shouldn't be an issue. We are
>>> missing the point of the Avocado cache and the possibility to manually
>>> add artifacts.
>> Sure but in this case the binaries are gone, you can't share them
>> and no
>
> Per https://mips.com/develop/tools/boot-loaders/:
> "To support existing users of these, and the QEMU project, YAMON is
> now available under the GPL License."

Maybe we should build it like the other ROMs?

Anyway I've dropped this commit from the PR so I'll leave it to you to
fix up however you want.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

[PATCH v2 5/7] tests/avocado: tag sbsa tests as tcg only

[Alex Bennée]

As firmware runs at a higher privilege level than the hypervisor we
can only run these tests under TCG emulation.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/avocado/machine_aarch64_sbsaref.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
index bdd1efc768..c25a691b64 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -20,6 +20,7 @@ class Aarch64SbsarefMachine(QemuSystemTest):
     """
     :avocado: tags=arch:aarch64
     :avocado: tags=machine:sbsa-ref
+    :avocado: tags=accel:tcg
     """
 
     timeout = 180
-- 
2.39.2

Re: [PATCH v2 5/7] tests/avocado: tag sbsa tests as tcg only

[Philippe Mathieu-Daudé]

On 1/12/23 10:36, Alex Bennée wrote:
> As firmware runs at a higher privilege level than the hypervisor we
> can only run these tests under TCG emulation.

Can we have this comment in the source please? Otherwise,
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   tests/avocado/machine_aarch64_sbsaref.py | 1 +
>   1 file changed, 1 insertion(+)

[PATCH v2 6/7] gitlab: build the correct microblaze target

[Alex Bennée]

We inadvertently built the LE target for BE tests.

Fixes: 78ebc00b06 (gitlab: shuffle some targets and reduce avocado noise)
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 .gitlab-ci.d/buildtest.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 7f9af83b10..62b5379a5e 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -41,7 +41,7 @@ build-system-ubuntu:
   variables:
     IMAGE: ubuntu2204
     CONFIGURE_ARGS: --enable-docs
-    TARGETS: alpha-softmmu microblazeel-softmmu mips64el-softmmu
+    TARGETS: alpha-softmmu microblaze-softmmu mips64el-softmmu
     MAKE_CHECK_ARGS: check-build
 
 check-system-ubuntu:
-- 
2.39.2

Re: [PATCH v2 6/7] gitlab: build the correct microblaze target

[Philippe Mathieu-Daudé]

On 1/12/23 10:36, Alex Bennée wrote:
> We inadvertently built the LE target for BE tests.
> 
> Fixes: 78ebc00b06 (gitlab: shuffle some targets and reduce avocado noise)
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   .gitlab-ci.d/buildtest.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)

Oops.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH v2 6/7] gitlab: build the correct microblaze target

[Thomas Huth]

On 01/12/2023 10.36, Alex Bennée wrote:
> We inadvertently built the LE target for BE tests.
> 
> Fixes: 78ebc00b06 (gitlab: shuffle some targets and reduce avocado noise)
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   .gitlab-ci.d/buildtest.yml | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 7f9af83b10..62b5379a5e 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -41,7 +41,7 @@ build-system-ubuntu:
>     variables:
>       IMAGE: ubuntu2204
>       CONFIGURE_ARGS: --enable-docs
> -    TARGETS: alpha-softmmu microblazeel-softmmu mips64el-softmmu
> +    TARGETS: alpha-softmmu microblaze-softmmu mips64el-softmmu
>       MAKE_CHECK_ARGS: check-build
>   
>   check-system-ubuntu:

We've got microblazeel-softmmu here and microblaze-softmmu in the 
build-system-fedora job. So please don't change the ubuntu job here, 
otherwise we're building the same target twice instead.

  Thomas

Re: [PATCH v2 6/7] gitlab: build the correct microblaze target

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> On 01/12/2023 10.36, Alex Bennée wrote:
>> We inadvertently built the LE target for BE tests.
>> Fixes: 78ebc00b06 (gitlab: shuffle some targets and reduce avocado
>> noise)
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>>   .gitlab-ci.d/buildtest.yml | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
>> index 7f9af83b10..62b5379a5e 100644
>> --- a/.gitlab-ci.d/buildtest.yml
>> +++ b/.gitlab-ci.d/buildtest.yml
>> @@ -41,7 +41,7 @@ build-system-ubuntu:
>>     variables:
>>       IMAGE: ubuntu2204
>>       CONFIGURE_ARGS: --enable-docs
>> -    TARGETS: alpha-softmmu microblazeel-softmmu mips64el-softmmu
>> +    TARGETS: alpha-softmmu microblaze-softmmu mips64el-softmmu
>>       MAKE_CHECK_ARGS: check-build
>>     check-system-ubuntu:
>
> We've got microblazeel-softmmu here and microblaze-softmmu in the
> build-system-fedora job. So please don't change the ubuntu job here,
> otherwise we're building the same target twice instead.

Hmm - what would be really useful is an actual microblazeel test image
so we can test what we build.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v2 6/7] gitlab: build the correct microblaze target

[Thomas Huth]

On 04/12/2023 13.40, Alex Bennée wrote:
> Thomas Huth <thuth@redhat.com> writes:
> 
>> On 01/12/2023 10.36, Alex Bennée wrote:
>>> We inadvertently built the LE target for BE tests.
>>> Fixes: 78ebc00b06 (gitlab: shuffle some targets and reduce avocado
>>> noise)
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> ---
>>>    .gitlab-ci.d/buildtest.yml | 2 +-
>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
>>> index 7f9af83b10..62b5379a5e 100644
>>> --- a/.gitlab-ci.d/buildtest.yml
>>> +++ b/.gitlab-ci.d/buildtest.yml
>>> @@ -41,7 +41,7 @@ build-system-ubuntu:
>>>      variables:
>>>        IMAGE: ubuntu2204
>>>        CONFIGURE_ARGS: --enable-docs
>>> -    TARGETS: alpha-softmmu microblazeel-softmmu mips64el-softmmu
>>> +    TARGETS: alpha-softmmu microblaze-softmmu mips64el-softmmu
>>>        MAKE_CHECK_ARGS: check-build
>>>      check-system-ubuntu:
>>
>> We've got microblazeel-softmmu here and microblaze-softmmu in the
>> build-system-fedora job. So please don't change the ubuntu job here,
>> otherwise we're building the same target twice instead.
> 
> Hmm - what would be really useful is an actual microblazeel test image
> so we can test what we build.

We've got at least a small test in tests/qtest/boot-serial-test.c, so we 
know that at least some instructions can be executed via TCG.

  Thomas

Re: [PATCH v2 6/7] gitlab: build the correct microblaze target

[Cédric Le Goater]

On 12/4/23 13:43, Thomas Huth wrote:
> On 04/12/2023 13.40, Alex Bennée wrote:
>> Thomas Huth <thuth@redhat.com> writes:
>>
>>> On 01/12/2023 10.36, Alex Bennée wrote:
>>>> We inadvertently built the LE target for BE tests.
>>>> Fixes: 78ebc00b06 (gitlab: shuffle some targets and reduce avocado
>>>> noise)
>>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>> ---
>>>>    .gitlab-ci.d/buildtest.yml | 2 +-
>>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
>>>> index 7f9af83b10..62b5379a5e 100644
>>>> --- a/.gitlab-ci.d/buildtest.yml
>>>> +++ b/.gitlab-ci.d/buildtest.yml
>>>> @@ -41,7 +41,7 @@ build-system-ubuntu:
>>>>      variables:
>>>>        IMAGE: ubuntu2204
>>>>        CONFIGURE_ARGS: --enable-docs
>>>> -    TARGETS: alpha-softmmu microblazeel-softmmu mips64el-softmmu
>>>> +    TARGETS: alpha-softmmu microblaze-softmmu mips64el-softmmu
>>>>        MAKE_CHECK_ARGS: check-build
>>>>      check-system-ubuntu:
>>>
>>> We've got microblazeel-softmmu here and microblaze-softmmu in the
>>> build-system-fedora job. So please don't change the ubuntu job here,
>>> otherwise we're building the same target twice instead.
>>
>> Hmm - what would be really useful is an actual microblazeel test image
>> so we can test what we build.
> 
> We've got at least a small test in tests/qtest/boot-serial-test.c, so we know that at least some instructions can be executed via TCG.

There are 2 configs under buildroot, qemu_microblazebe_mmu_defconfig and
qemu_microblazeel_mmu_defconfig, we could possibly use.

C.

Re: [PATCH v2 6/7] gitlab: build the correct microblaze target

[Thomas Huth]

On 04/12/2023 13.48, Cédric Le Goater wrote:
> On 12/4/23 13:43, Thomas Huth wrote:
>> On 04/12/2023 13.40, Alex Bennée wrote:
>>> Thomas Huth <thuth@redhat.com> writes:
>>>
>>>> On 01/12/2023 10.36, Alex Bennée wrote:
>>>>> We inadvertently built the LE target for BE tests.
>>>>> Fixes: 78ebc00b06 (gitlab: shuffle some targets and reduce avocado
>>>>> noise)
>>>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>>> ---
>>>>>    .gitlab-ci.d/buildtest.yml | 2 +-
>>>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>>> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
>>>>> index 7f9af83b10..62b5379a5e 100644
>>>>> --- a/.gitlab-ci.d/buildtest.yml
>>>>> +++ b/.gitlab-ci.d/buildtest.yml
>>>>> @@ -41,7 +41,7 @@ build-system-ubuntu:
>>>>>      variables:
>>>>>        IMAGE: ubuntu2204
>>>>>        CONFIGURE_ARGS: --enable-docs
>>>>> -    TARGETS: alpha-softmmu microblazeel-softmmu mips64el-softmmu
>>>>> +    TARGETS: alpha-softmmu microblaze-softmmu mips64el-softmmu
>>>>>        MAKE_CHECK_ARGS: check-build
>>>>>      check-system-ubuntu:
>>>>
>>>> We've got microblazeel-softmmu here and microblaze-softmmu in the
>>>> build-system-fedora job. So please don't change the ubuntu job here,
>>>> otherwise we're building the same target twice instead.
>>>
>>> Hmm - what would be really useful is an actual microblazeel test image
>>> so we can test what we build.
>>
>> We've got at least a small test in tests/qtest/boot-serial-test.c, so we 
>> know that at least some instructions can be executed via TCG.
> 
> There are 2 configs under buildroot, qemu_microblazebe_mmu_defconfig and
> qemu_microblazeel_mmu_defconfig, we could possibly use.

I already used the big endian config for going shopping for ballerinas 
(which we already use in one of the avocado tests):

  http://www.qemu-advent-calendar.org/2018/#day-17

And as I just learnt today, the QEMU advent calendar opens its doors again 
in 2023:

  http://www.qemu-advent-calendar.org/2023/

Let's see, maybe we can have a nice surprise for microblazeel this time...

  Thomas

[PATCH v2 7/7] gitlab: add optional job to run flaky avocado tests

[Alex Bennée]

One problem with flaky tests is they often only fail under CI
conditions which makes it hard to debug. We add an optional allow_fail
job so developers can trigger the only the flaky tests in the CI
environment if they are debugging.

Message-Id: <20231130153333.2424775-4-alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

---
v2
  - add missing ppc64/aarch64 build
---
 docs/devel/testing.rst                   | 15 ++++++------
 .gitlab-ci.d/buildtest.yml               | 30 ++++++++++++++++++++++++
 tests/avocado/boot_linux.py              |  2 ++
 tests/avocado/boot_linux_console.py      |  1 +
 tests/avocado/intel_iommu.py             |  1 +
 tests/avocado/linux_initrd.py            |  2 ++
 tests/avocado/machine_aspeed.py          |  2 ++
 tests/avocado/machine_mips_malta.py      |  2 ++
 tests/avocado/machine_rx_gdbsim.py       |  2 ++
 tests/avocado/machine_s390_ccw_virtio.py |  1 +
 tests/avocado/replay_kernel.py           |  2 ++
 tests/avocado/reverse_debugging.py       |  2 ++
 tests/avocado/smmu.py                    |  1 +
 tests/avocado/tuxrun_baselines.py        |  1 +
 14 files changed, 57 insertions(+), 7 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 76465b8f3d..bd132306c1 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -1387,16 +1387,17 @@ the code snippet below:
   def test(self):
       do_something()
 
-Tests should not live in this state forever and should either be fixed
-or eventually removed.
-
-To run such tests locally you will need to set the environment
-variable. For example:
+You can also add ``:avocado: tags=flaky`` to the test meta-data so
+only the flaky tests can be run as a group:
 
 .. code::
 
-   env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado run \
-      tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg
+   env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado \
+      run tests/avocado -filter-by-tags=flaky
+
+Tests should not live in this state forever and should either be fixed
+or eventually removed.
+
 
 Uninstalling Avocado
 ~~~~~~~~~~~~~~~~~~~~
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 62b5379a5e..91663946de 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -217,6 +217,36 @@ avocado-system-opensuse:
     MAKE_CHECK_ARGS: check-avocado
     AVOCADO_TAGS: arch:s390x arch:x86_64 arch:aarch64
 
+#
+# Flaky tests. We don't run these by default and they are allow fail
+# but often the CI system is the only way to trigger the failures.
+#
+
+build-system-flaky:
+  extends:
+    - .native_build_job_template
+    - .native_build_artifact_template
+  needs:
+    job: amd64-debian-container
+  variables:
+    IMAGE: debian
+    QEMU_JOB_OPTIONAL: 1
+    TARGETS: aarch64-softmmu arm-softmmu mips64el-softmmu
+      ppc64-softmmu rx-softmmu s390x-softmmu sh4-softmmu x86_64-softmmu
+    MAKE_CHECK_ARGS: check-build
+
+avocado-system-flaky:
+  extends: .avocado_test_job_template
+  needs:
+    - job: build-system-flaky
+      artifacts: true
+  allow_failure: true
+  variables:
+    IMAGE: debian
+    MAKE_CHECK_ARGS: check-avocado
+    QEMU_JOB_OPTIONAL: 1
+    QEMU_TEST_FLAKY_TESTS: 1
+    AVOCADO_TAGS: flaky
 
 # This jobs explicitly disable TCG (--disable-tcg), KVM is detected by
 # the configure script. The container doesn't contain Xen headers so
diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py
index 9e9773e6e1..7c4769904e 100644
--- a/tests/avocado/boot_linux.py
+++ b/tests/avocado/boot_linux.py
@@ -99,6 +99,7 @@ def test_pseries_tcg(self):
         """
         :avocado: tags=machine:pseries
         :avocado: tags=accel:tcg
+        :avocado: tags=flaky
         """
         self.require_accelerator("tcg")
         self.vm.add_args("-accel", "tcg")
@@ -118,6 +119,7 @@ def test_s390_ccw_virtio_tcg(self):
         """
         :avocado: tags=machine:s390-ccw-virtio
         :avocado: tags=accel:tcg
+        :avocado: tags=flaky
         """
         self.require_accelerator("tcg")
         self.vm.add_args("-accel", "tcg")
diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
index 5d978f6dd0..cffa4da9f0 100644
--- a/tests/avocado/boot_linux_console.py
+++ b/tests/avocado/boot_linux_console.py
@@ -1403,6 +1403,7 @@ def test_sh4_r2d(self):
         """
         :avocado: tags=arch:sh4
         :avocado: tags=machine:r2d
+        :avocado: tags=flaky
         """
         tar_hash = 'fe06a4fd8ccbf2e27928d64472939d47829d4c7e'
         self.vm.add_args('-append', 'console=ttySC1')
diff --git a/tests/avocado/intel_iommu.py b/tests/avocado/intel_iommu.py
index 2dd11a6346..f04ee1cf9d 100644
--- a/tests/avocado/intel_iommu.py
+++ b/tests/avocado/intel_iommu.py
@@ -22,6 +22,7 @@ class IntelIOMMU(LinuxTest):
     :avocado: tags=machine:q35
     :avocado: tags=accel:kvm
     :avocado: tags=intel_iommu
+    :avocado: tags=flaky
     """
 
     IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'
diff --git a/tests/avocado/linux_initrd.py b/tests/avocado/linux_initrd.py
index c40a987bd1..aad5b19bd9 100644
--- a/tests/avocado/linux_initrd.py
+++ b/tests/avocado/linux_initrd.py
@@ -57,6 +57,8 @@ def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self):
 
     def test_with_2gib_file_should_work_with_linux_v4_16(self):
         """
+        :avocado: tags=flaky
+
         QEMU has supported up to 4 GiB initrd for recent kernel
         Expect guest can reach 'Unpacking initramfs...'
         """
diff --git a/tests/avocado/machine_aspeed.py b/tests/avocado/machine_aspeed.py
index 258fb50c47..6fa5459a07 100644
--- a/tests/avocado/machine_aspeed.py
+++ b/tests/avocado/machine_aspeed.py
@@ -317,6 +317,7 @@ def test_arm_ast2500_evb_sdk(self):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:ast2500-evb
+        :avocado: tags=flaky
         """
 
         image_url = ('https://github.com/AspeedTech-BMC/openbmc/releases/'
@@ -336,6 +337,7 @@ def test_arm_ast2600_evb_sdk(self):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:ast2600-evb
+        :avocado: tags=flaky
         """
 
         image_url = ('https://github.com/AspeedTech-BMC/openbmc/releases/'
diff --git a/tests/avocado/machine_mips_malta.py b/tests/avocado/machine_mips_malta.py
index 3a1ec85c20..9c997afeb0 100644
--- a/tests/avocado/machine_mips_malta.py
+++ b/tests/avocado/machine_mips_malta.py
@@ -109,6 +109,7 @@ def test_mips_malta_i6400_framebuffer_logo_7cores(self):
         :avocado: tags=machine:malta
         :avocado: tags=cpu:I6400
         :avocado: tags=mips:smp
+        :avocado: tags=flaky
         """
         self.do_test_i6400_framebuffer_logo(7)
 
@@ -120,5 +121,6 @@ def test_mips_malta_i6400_framebuffer_logo_8cores(self):
         :avocado: tags=machine:malta
         :avocado: tags=cpu:I6400
         :avocado: tags=mips:smp
+        :avocado: tags=flaky
         """
         self.do_test_i6400_framebuffer_logo(8)
diff --git a/tests/avocado/machine_rx_gdbsim.py b/tests/avocado/machine_rx_gdbsim.py
index 350a73fbbb..412a7a5089 100644
--- a/tests/avocado/machine_rx_gdbsim.py
+++ b/tests/avocado/machine_rx_gdbsim.py
@@ -31,6 +31,7 @@ def test_uboot(self):
         :avocado: tags=arch:rx
         :avocado: tags=machine:gdbsim-r5f562n8
         :avocado: tags=endian:little
+        :avocado: tags=flaky
         """
         uboot_url = ('https://acc.dl.osdn.jp/users/23/23888/u-boot.bin.gz')
         uboot_hash = '9b78dbd43b40b2526848c0b1ce9de02c24f4dcdb'
@@ -56,6 +57,7 @@ def test_linux_sash(self):
         :avocado: tags=arch:rx
         :avocado: tags=machine:gdbsim-r5f562n7
         :avocado: tags=endian:little
+        :avocado: tags=flaky
         """
         dtb_url = ('https://acc.dl.osdn.jp/users/23/23887/rx-virt.dtb')
         dtb_hash = '7b4e4e2c71905da44e86ce47adee2210b026ac18'
diff --git a/tests/avocado/machine_s390_ccw_virtio.py b/tests/avocado/machine_s390_ccw_virtio.py
index 61e75d8f9e..26e938c9e9 100644
--- a/tests/avocado/machine_s390_ccw_virtio.py
+++ b/tests/avocado/machine_s390_ccw_virtio.py
@@ -167,6 +167,7 @@ def test_s390x_fedora(self):
         :avocado: tags=device:virtio-gpu
         :avocado: tags=device:virtio-crypto
         :avocado: tags=device:virtio-net
+        :avocado: tags=flaky
         """
 
         kernel_url = ('https://archives.fedoraproject.org/pub/archive'
diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py
index 0d32cc280e..af086eab08 100644
--- a/tests/avocado/replay_kernel.py
+++ b/tests/avocado/replay_kernel.py
@@ -88,6 +88,7 @@ def test_x86_64_pc(self):
         """
         :avocado: tags=arch:x86_64
         :avocado: tags=machine:pc
+        :avocado: tags=flaky
         """
         kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
                       '/linux/releases/29/Everything/x86_64/os/images/pxeboot'
@@ -186,6 +187,7 @@ def test_arm_cubieboard_initrd(self):
         """
         :avocado: tags=arch:arm
         :avocado: tags=machine:cubieboard
+        :avocado: tags=flaky
         """
         deb_url = ('https://apt.armbian.com/pool/main/l/'
                    'linux-5.10.16-sunxi/linux-image-current-sunxi_21.02.2_armhf.deb')
diff --git a/tests/avocado/reverse_debugging.py b/tests/avocado/reverse_debugging.py
index 9a468321e5..4cce5a5598 100644
--- a/tests/avocado/reverse_debugging.py
+++ b/tests/avocado/reverse_debugging.py
@@ -255,6 +255,7 @@ def test_ppc64_pseries(self):
         """
         :avocado: tags=arch:ppc64
         :avocado: tags=machine:pseries
+        :avocado: tags=flaky
         """
         # SLOF branches back to its entry point, which causes this test
         # to take the 'hit a breakpoint again' path. That's not a problem,
@@ -269,6 +270,7 @@ def test_ppc64_powernv(self):
         """
         :avocado: tags=arch:ppc64
         :avocado: tags=machine:powernv
+        :avocado: tags=flaky
         """
         self.endian_is_le = False
         self.reverse_debugging()
diff --git a/tests/avocado/smmu.py b/tests/avocado/smmu.py
index 05b34418a5..21ff030ca7 100644
--- a/tests/avocado/smmu.py
+++ b/tests/avocado/smmu.py
@@ -22,6 +22,7 @@ class SMMU(LinuxTest):
     :avocado: tags=machine:virt
     :avocado: tags=distro:fedora
     :avocado: tags=smmu
+    :avocado: tags=flaky
     """
 
     IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'
diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py
index 5f859f4e6f..a936a3b780 100644
--- a/tests/avocado/tuxrun_baselines.py
+++ b/tests/avocado/tuxrun_baselines.py
@@ -561,6 +561,7 @@ def test_sh4(self):
         :avocado: tags=image:zImage
         :avocado: tags=root:sda
         :avocado: tags=console:ttySC1
+        :avocado: tags=flaky
         """
         sums = { "rootfs.ext4.zst" :
                  "3592a7a3d5a641e8b9821449e77bc43c9904a56c30d45da0694349cfd86743fd",
-- 
2.39.2

Re: [PATCH v2 7/7] gitlab: add optional job to run flaky avocado tests

[Philippe Mathieu-Daudé]

On 1/12/23 10:36, Alex Bennée wrote:
> One problem with flaky tests is they often only fail under CI
> conditions which makes it hard to debug. We add an optional allow_fail
> job so developers can trigger the only the flaky tests in the CI
> environment if they are debugging.
> 
> Message-Id: <20231130153333.2424775-4-alex.bennee@linaro.org>
> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> 
> ---
> v2
>    - add missing ppc64/aarch64 build
> ---
>   docs/devel/testing.rst                   | 15 ++++++------
>   .gitlab-ci.d/buildtest.yml               | 30 ++++++++++++++++++++++++

This should be split, first add the 'flaky' infrastructure, then
mark tests as flaky, referencing the GitLab issue URL.

>   tests/avocado/boot_linux.py              |  2 ++
>   tests/avocado/boot_linux_console.py      |  1 +
>   tests/avocado/intel_iommu.py             |  1 +
>   tests/avocado/linux_initrd.py            |  2 ++
>   tests/avocado/machine_aspeed.py          |  2 ++
>   tests/avocado/machine_mips_malta.py      |  2 ++
>   tests/avocado/machine_rx_gdbsim.py       |  2 ++
>   tests/avocado/machine_s390_ccw_virtio.py |  1 +
>   tests/avocado/replay_kernel.py           |  2 ++
>   tests/avocado/reverse_debugging.py       |  2 ++
>   tests/avocado/smmu.py                    |  1 +
>   tests/avocado/tuxrun_baselines.py        |  1 +
>   14 files changed, 57 insertions(+), 7 deletions(-)

Re: [PATCH v2 0/7] final fixes for 8.2

[Stefan Hajnoczi]

On Fri, 1 Dec 2023 at 07:56, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> 8.2 is looking fairly stable but I do have one bug fix for gdbstub
> which I came across while debugging something else. The changes for
> avocado rationalise all flaky tests under the QEMU_TEST_FLAKY_TESTS
> environment variable. The final patch re-adds the flaky tests to the
> CI as a manually run allow_fail job so we can still attempt to debug
> their failure in the place they tend to fall over.

Acked-by: Stefan Hajnoczi <stefanha@redhat.com>

>
> v2
> --
>
>   - addressed some review comments
>   - emphasised raising a bug for failing tests
>   - drop some tests now missing assets online
>   - fix wrong microblaze target for tests
>
> The following still need review:
>
>   gitlab: build the correct microblaze target
>   tests/avocado: tag sbsa tests as tcg only
>   tests/avocado: drop malta yamon tests
>   tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test
>
> Alex.
>
> Alex Bennée (7):
>   gdbstub: use a better signal when we halt for IO reasons
>   docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS
>   tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test
>   tests/avocado: drop malta yamon tests
>   tests/avocado: tag sbsa tests as tcg only
>   gitlab: build the correct microblaze target
>   gitlab: add optional job to run flaky avocado tests
>
>  docs/devel/testing.rst                   | 32 ++++++++++------
>  gdbstub/internals.h                      |  1 +
>  gdbstub/system.c                         |  2 +-
>  .gitlab-ci.d/buildtest.yml               | 32 +++++++++++++++-
>  tests/avocado/boot_linux.py              | 10 +++--
>  tests/avocado/boot_linux_console.py      | 28 ++------------
>  tests/avocado/intel_iommu.py             |  6 ++-
>  tests/avocado/linux_initrd.py            |  7 +++-
>  tests/avocado/machine_aarch64_sbsaref.py |  1 +
>  tests/avocado/machine_aspeed.py          | 10 +++--
>  tests/avocado/machine_mips_malta.py      | 47 ++++--------------------
>  tests/avocado/machine_rx_gdbsim.py       | 10 +++--
>  tests/avocado/machine_s390_ccw_virtio.py |  3 +-
>  tests/avocado/replay_kernel.py           |  8 +++-
>  tests/avocado/reverse_debugging.py       | 16 +++++---
>  tests/avocado/smmu.py                    |  6 ++-
>  tests/avocado/tuxrun_baselines.py        |  5 ++-
>  17 files changed, 122 insertions(+), 102 deletions(-)
>
> --
> 2.39.2
>
>