* [PULL 1/6] gdbstub: use a better signal when we halt for IO reasons
2023-12-01 15:12 [PULL 0/6] gdbstub, avocado and gitlab updates Alex Bennée
@ 2023-12-01 15:12 ` Alex Bennée
2023-12-01 15:12 ` [PULL 2/6] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS Alex Bennée
` (5 subsequent siblings)
6 siblings, 0 replies; 12+ messages in thread
From: Alex Bennée @ 2023-12-01 15:12 UTC (permalink / raw)
To: qemu-devel
Cc: Alex Bennée, Luis Machado, Richard Henderson,
Stefan Hajnoczi, Philippe Mathieu-Daudé
The gdb description GDB_SIGNAL_IO is "I/O possible" and by default gdb
will try and restart the guest, getting us nowhere. Report
GDB_SIGNAL_STOP instead which should at least halt the session at the
failure point.
Reviewed-by: Luis Machado <luis.machado@arm.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231201093633.2551497-2-alex.bennee@linaro.org>
diff --git a/gdbstub/internals.h b/gdbstub/internals.h
index 465c24b36e..5c0c725e54 100644
--- a/gdbstub/internals.h
+++ b/gdbstub/internals.h
@@ -24,6 +24,7 @@ enum {
GDB_SIGNAL_TRAP = 5,
GDB_SIGNAL_ABRT = 6,
GDB_SIGNAL_ALRM = 14,
+ GDB_SIGNAL_STOP = 17,
GDB_SIGNAL_IO = 23,
GDB_SIGNAL_XCPU = 24,
GDB_SIGNAL_UNKNOWN = 143
diff --git a/gdbstub/system.c b/gdbstub/system.c
index 783ac140b9..83fd452800 100644
--- a/gdbstub/system.c
+++ b/gdbstub/system.c
@@ -183,7 +183,7 @@ static void gdb_vm_state_change(void *opaque, bool running, RunState state)
break;
case RUN_STATE_IO_ERROR:
trace_gdbstub_hit_io_error();
- ret = GDB_SIGNAL_IO;
+ ret = GDB_SIGNAL_STOP;
break;
case RUN_STATE_WATCHDOG:
trace_gdbstub_hit_watchdog();
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PULL 2/6] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS
2023-12-01 15:12 [PULL 0/6] gdbstub, avocado and gitlab updates Alex Bennée
2023-12-01 15:12 ` [PULL 1/6] gdbstub: use a better signal when we halt for IO reasons Alex Bennée
@ 2023-12-01 15:12 ` Alex Bennée
2023-12-01 15:12 ` [PULL 3/6] tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test Alex Bennée
` (4 subsequent siblings)
6 siblings, 0 replies; 12+ messages in thread
From: Alex Bennée @ 2023-12-01 15:12 UTC (permalink / raw)
To: qemu-devel
Cc: Alex Bennée, Philippe Mathieu-Daudé, Richard Henderson,
Stefan Hajnoczi, Thomas Huth, Wainer dos Santos Moschetta,
Beraldo Leal, Cleber Rosa, Cédric Le Goater, Peter Maydell,
Andrew Jeffery, Joel Stanley, Aurelien Jarno, Yoshinori Sato,
Halil Pasic, Christian Borntraeger, Eric Farman, Pavel Dovgalyuk,
Paolo Bonzini, Eric Auger, open list:ASPEED BMCs,
open list:S390 Virtio-ccw
It doesn't make sense to have two classes of flaky tests. While it may
take the constrained environment of CI to trigger failures easily it
doesn't mean they don't occasionally happen on developer machines. As
CI is the gating factor to passing there is no point developers
running the tests locally anyway unless they are trying to fix things.
While we are at it update the language in the docs to discourage the
QEMU_TEST_FLAKY_TESTS becoming a permanent solution.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231201093633.2551497-3-alex.bennee@linaro.org>
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 22218dbedb..76465b8f3d 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -1371,23 +1371,32 @@ conditions. For example, tests that take longer to execute when QEMU is
compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable
has been used to determine whether those tests should run or not.
-GITLAB_CI
-^^^^^^^^^
-A number of tests are flagged to not run on the GitLab CI. Usually because
-they proved to the flaky or there are constraints on the CI environment which
-would make them fail. If you encounter a similar situation then use that
-variable as shown on the code snippet below to skip the test:
+QEMU_TEST_FLAKY_TESTS
+^^^^^^^^^^^^^^^^^^^^^
+Some tests are not working reliably and thus are disabled by default.
+This includes tests that don't run reliably on GitLab's CI which
+usually expose real issues that are rarely seen on developer machines
+due to the constraints of the CI environment. If you encounter a
+similar situation then raise a bug and then mark the test as shown on
+the code snippet below:
.. code::
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ # See https://gitlab.com/qemu-project/qemu/-/issues/nnnn
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test(self):
do_something()
-QEMU_TEST_FLAKY_TESTS
-^^^^^^^^^^^^^^^^^^^^^
-Some tests are not working reliably and thus are disabled by default.
-Set this environment variable to enable them.
+Tests should not live in this state forever and should either be fixed
+or eventually removed.
+
+To run such tests locally you will need to set the environment
+variable. For example:
+
+.. code::
+
+ env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado run \
+ tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg
Uninstalling Avocado
~~~~~~~~~~~~~~~~~~~~
diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py
index be30dcbd58..9e9773e6e1 100644
--- a/tests/avocado/boot_linux.py
+++ b/tests/avocado/boot_linux.py
@@ -12,7 +12,7 @@
from avocado_qemu import LinuxTest, BUILD_DIR
-from avocado import skipIf
+from avocado import skipUnless
class BootLinuxX8664(LinuxTest):
@@ -93,7 +93,8 @@ class BootLinuxPPC64(LinuxTest):
timeout = 360
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_pseries_tcg(self):
"""
:avocado: tags=machine:pseries
@@ -111,7 +112,8 @@ class BootLinuxS390X(LinuxTest):
timeout = 240
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_s390_ccw_virtio_tcg(self):
"""
:avocado: tags=machine:s390-ccw-virtio
diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
index 6eab515718..231b4f68e5 100644
--- a/tests/avocado/boot_linux_console.py
+++ b/tests/avocado/boot_linux_console.py
@@ -15,7 +15,7 @@
from avocado import skip
from avocado import skipUnless
-from avocado import skipIf
+from avocado import skipUnless
from avocado_qemu import QemuSystemTest
from avocado_qemu import exec_command
from avocado_qemu import exec_command_and_wait_for_pattern
@@ -1419,7 +1419,8 @@ def test_ppc_mac99(self):
# This test has a 6-10% failure rate on various hosts that look
# like issues with a buggy kernel. As a result we don't want it
# gating releases on Gitlab.
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_sh4_r2d(self):
"""
:avocado: tags=arch:sh4
diff --git a/tests/avocado/intel_iommu.py b/tests/avocado/intel_iommu.py
index 77635ab56c..2dd11a6346 100644
--- a/tests/avocado/intel_iommu.py
+++ b/tests/avocado/intel_iommu.py
@@ -9,10 +9,11 @@
# later. See the COPYING file in the top-level directory.
import os
-from avocado import skipIf
+from avocado import skipUnless
from avocado_qemu import LinuxTest
-@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
class IntelIOMMU(LinuxTest):
"""
:avocado: tags=arch:x86_64
diff --git a/tests/avocado/linux_initrd.py b/tests/avocado/linux_initrd.py
index ba02e5a563..c40a987bd1 100644
--- a/tests/avocado/linux_initrd.py
+++ b/tests/avocado/linux_initrd.py
@@ -13,7 +13,7 @@
import tempfile
from avocado_qemu import QemuSystemTest
-from avocado import skipIf
+from avocado import skipUnless
class LinuxInitrd(QemuSystemTest):
@@ -53,7 +53,8 @@ def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self):
max_size + 1)
self.assertRegex(self.vm.get_log(), expected_msg)
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_with_2gib_file_should_work_with_linux_v4_16(self):
"""
QEMU has supported up to 4 GiB initrd for recent kernel
diff --git a/tests/avocado/machine_aspeed.py b/tests/avocado/machine_aspeed.py
index df31b2a8a8..258fb50c47 100644
--- a/tests/avocado/machine_aspeed.py
+++ b/tests/avocado/machine_aspeed.py
@@ -18,7 +18,7 @@
from avocado_qemu import interrupt_interactive_console_until_pattern
from avocado_qemu import has_cmd
from avocado.utils import archive
-from avocado import skipIf
+from avocado import skipUnless
from avocado import skipUnless
@@ -311,7 +311,8 @@ def do_test_arm_aspeed_sdk_start(self, image):
self, 'boot', '## Loading kernel from FIT Image')
self.wait_for_console_pattern('Starting kernel ...')
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_arm_ast2500_evb_sdk(self):
"""
:avocado: tags=arch:arm
@@ -329,7 +330,8 @@ def test_arm_ast2500_evb_sdk(self):
self.workdir + '/ast2500-default/image-bmc')
self.wait_for_console_pattern('nodistro.0 ast2500-default ttyS4')
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_arm_ast2600_evb_sdk(self):
"""
:avocado: tags=arch:arm
diff --git a/tests/avocado/machine_mips_malta.py b/tests/avocado/machine_mips_malta.py
index 9bd54518bf..959dcf5602 100644
--- a/tests/avocado/machine_mips_malta.py
+++ b/tests/avocado/machine_mips_malta.py
@@ -11,7 +11,7 @@
import gzip
import logging
-from avocado import skipIf
+from avocado import skipUnless
from avocado import skipUnless
from avocado.utils import archive
from avocado_qemu import QemuSystemTest
@@ -101,7 +101,8 @@ def test_mips_malta_i6400_framebuffer_logo_1core(self):
"""
self.do_test_i6400_framebuffer_logo(1)
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_mips_malta_i6400_framebuffer_logo_7cores(self):
"""
:avocado: tags=arch:mips64el
@@ -111,7 +112,8 @@ def test_mips_malta_i6400_framebuffer_logo_7cores(self):
"""
self.do_test_i6400_framebuffer_logo(7)
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_mips_malta_i6400_framebuffer_logo_8cores(self):
"""
:avocado: tags=arch:mips64el
diff --git a/tests/avocado/machine_rx_gdbsim.py b/tests/avocado/machine_rx_gdbsim.py
index 6cd8704b01..350a73fbbb 100644
--- a/tests/avocado/machine_rx_gdbsim.py
+++ b/tests/avocado/machine_rx_gdbsim.py
@@ -10,7 +10,7 @@
import os
-from avocado import skipIf
+from avocado import skipUnless
from avocado_qemu import QemuSystemTest
from avocado_qemu import exec_command_and_wait_for_pattern
from avocado_qemu import wait_for_console_pattern
@@ -22,7 +22,8 @@ class RxGdbSimMachine(QemuSystemTest):
timeout = 30
KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_uboot(self):
"""
U-Boot and checks that the console is operational.
@@ -46,7 +47,8 @@ def test_uboot(self):
# FIXME limit baudrate on chardev, else we type too fast
#exec_command_and_wait_for_pattern(self, 'version', gcc_version)
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_linux_sash(self):
"""
Boots a Linux kernel and checks that the console is operational.
diff --git a/tests/avocado/machine_s390_ccw_virtio.py b/tests/avocado/machine_s390_ccw_virtio.py
index ffd914ded9..61e75d8f9e 100644
--- a/tests/avocado/machine_s390_ccw_virtio.py
+++ b/tests/avocado/machine_s390_ccw_virtio.py
@@ -12,7 +12,7 @@
import os
import tempfile
-from avocado import skipIf
+from avocado import skipUnless
from avocado_qemu import QemuSystemTest
from avocado_qemu import exec_command_and_wait_for_pattern
from avocado_qemu import wait_for_console_pattern
diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py
index 53cb7e5091..0d32cc280e 100644
--- a/tests/avocado/replay_kernel.py
+++ b/tests/avocado/replay_kernel.py
@@ -15,7 +15,7 @@
import time
from avocado import skip
-from avocado import skipIf
+from avocado import skipUnless
from avocado import skipUnless
from avocado_qemu import wait_for_console_pattern
from avocado.utils import archive
@@ -82,6 +82,7 @@ def run_rr(self, kernel_path, kernel_command_line, console_pattern,
class ReplayKernelNormal(ReplayKernelBase):
+ # See https://gitlab.com/qemu-project/qemu/-/issues/2010
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test sometimes gets stuck')
def test_x86_64_pc(self):
"""
@@ -179,7 +180,8 @@ def test_arm_virt(self):
self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1)
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_arm_cubieboard_initrd(self):
"""
:avocado: tags=arch:arm
diff --git a/tests/avocado/reverse_debugging.py b/tests/avocado/reverse_debugging.py
index ed04e92bb4..9a468321e5 100644
--- a/tests/avocado/reverse_debugging.py
+++ b/tests/avocado/reverse_debugging.py
@@ -10,7 +10,7 @@
import os
import logging
-from avocado import skipIf
+from avocado import skipUnless
from avocado_qemu import BUILD_DIR
from avocado.utils import datadrainer
from avocado.utils import gdb
@@ -206,7 +206,8 @@ def get_pc(self, g):
+ self.get_reg_le(g, self.REG_CS) * 0x10
# unidentified gitlab timeout problem
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_x86_64_pc(self):
"""
:avocado: tags=arch:x86_64
@@ -223,7 +224,8 @@ class ReverseDebugging_AArch64(ReverseDebugging):
REG_PC = 32
# unidentified gitlab timeout problem
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_aarch64_virt(self):
"""
:avocado: tags=arch:aarch64
@@ -247,7 +249,8 @@ class ReverseDebugging_ppc64(ReverseDebugging):
REG_PC = 0x40
# unidentified gitlab timeout problem
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_ppc64_pseries(self):
"""
:avocado: tags=arch:ppc64
@@ -260,7 +263,8 @@ def test_ppc64_pseries(self):
self.reverse_debugging()
# See https://gitlab.com/qemu-project/qemu/-/issues/1992
- @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
def test_ppc64_powernv(self):
"""
:avocado: tags=arch:ppc64
diff --git a/tests/avocado/smmu.py b/tests/avocado/smmu.py
index b3c4de6bf4..05b34418a5 100644
--- a/tests/avocado/smmu.py
+++ b/tests/avocado/smmu.py
@@ -9,10 +9,11 @@
# later. See the COPYING file in the top-level directory.
import os
-from avocado import skipIf
+from avocado import skipUnless
from avocado_qemu import LinuxTest, BUILD_DIR
-@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
+@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
+
class SMMU(LinuxTest):
"""
:avocado: tags=accel:kvm
diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py
index c99bea6c0b..5f859f4e6f 100644
--- a/tests/avocado/tuxrun_baselines.py
+++ b/tests/avocado/tuxrun_baselines.py
@@ -13,7 +13,7 @@
import time
import tempfile
-from avocado import skip, skipIf
+from avocado import skip, skipUnless
from avocado_qemu import QemuSystemTest
from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
from avocado_qemu import wait_for_console_pattern
@@ -551,7 +551,7 @@ def test_s390(self):
haltmsg="Requesting system halt")
# Note: some segfaults caused by unaligned userspace access
- @skipIf(os.getenv('GITLAB_CI'), 'Skipping unstable test on GitLab')
+ @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test_sh4(self):
"""
:avocado: tags=arch:sh4
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PULL 3/6] tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test
2023-12-01 15:12 [PULL 0/6] gdbstub, avocado and gitlab updates Alex Bennée
2023-12-01 15:12 ` [PULL 1/6] gdbstub: use a better signal when we halt for IO reasons Alex Bennée
2023-12-01 15:12 ` [PULL 2/6] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS Alex Bennée
@ 2023-12-01 15:12 ` Alex Bennée
2023-12-01 16:49 ` Philippe Mathieu-Daudé
2023-12-01 15:12 ` [PULL 4/6] tests/avocado: tag sbsa tests as tcg only Alex Bennée
` (3 subsequent siblings)
6 siblings, 1 reply; 12+ messages in thread
From: Alex Bennée @ 2023-12-01 15:12 UTC (permalink / raw)
To: qemu-devel
Cc: Alex Bennée, Stefan Hajnoczi, Philippe Mathieu-Daudé,
Cleber Rosa, Wainer dos Santos Moschetta, Beraldo Leal,
Jiaxun Yang
The assets are no longer archived by Debian so we can't run this on
CI. While some people may still have the test in their cache we do
have more recent images from tuxrun so this isn't a great loss.
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231201093633.2551497-4-alex.bennee@linaro.org>
diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
index 231b4f68e5..ba2f99b53a 100644
--- a/tests/avocado/boot_linux_console.py
+++ b/tests/avocado/boot_linux_console.py
@@ -191,47 +191,6 @@ def test_mips64el_fuloong2e(self):
console_pattern = 'Kernel command line: %s' % kernel_command_line
self.wait_for_console_pattern(console_pattern)
- def test_mips_malta_cpio(self):
- """
- :avocado: tags=arch:mips
- :avocado: tags=machine:malta
- :avocado: tags=endian:big
- """
- deb_url = ('http://snapshot.debian.org/archive/debian/'
- '20160601T041800Z/pool/main/l/linux/'
- 'linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb')
- deb_hash = 'a3c84f3e88b54e06107d65a410d1d1e8e0f340f8'
- deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
- kernel_path = self.extract_from_deb(deb_path,
- '/boot/vmlinux-4.5.0-2-4kc-malta')
- initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
- '8584a59ed9e5eb5ee7ca91f6d74bbb06619205b8/rootfs/'
- 'mips/rootfs.cpio.gz')
- initrd_hash = 'bf806e17009360a866bf537f6de66590de349a99'
- initrd_path_gz = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
- initrd_path = self.workdir + "rootfs.cpio"
- archive.gzip_uncompress(initrd_path_gz, initrd_path)
-
- self.vm.set_console()
- kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
- + 'console=ttyS0 console=tty '
- + 'rdinit=/sbin/init noreboot')
- self.vm.add_args('-kernel', kernel_path,
- '-initrd', initrd_path,
- '-append', kernel_command_line,
- '-no-reboot')
- self.vm.launch()
- self.wait_for_console_pattern('Boot successful.')
-
- exec_command_and_wait_for_pattern(self, 'cat /proc/cpuinfo',
- 'BogoMIPS')
- exec_command_and_wait_for_pattern(self, 'uname -a',
- 'Debian')
- exec_command_and_wait_for_pattern(self, 'reboot',
- 'reboot: Restarting system')
- # Wait for VM to shut down gracefully
- self.vm.wait()
-
@skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
def test_mips64el_malta_5KEc_cpio(self):
"""
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PULL 3/6] tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test
2023-12-01 15:12 ` [PULL 3/6] tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test Alex Bennée
@ 2023-12-01 16:49 ` Philippe Mathieu-Daudé
2023-12-01 17:46 ` Alex Bennée
0 siblings, 1 reply; 12+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-12-01 16:49 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Cc: Stefan Hajnoczi, Cleber Rosa, Wainer dos Santos Moschetta,
Beraldo Leal, Jiaxun Yang
On 1/12/23 16:12, Alex Bennée wrote:
> The assets are no longer archived by Debian so we can't run this on
> CI. While some people may still have the test in their cache we do
> have more recent images from tuxrun so this isn't a great loss.
>
> Cc: Stefan Hajnoczi <stefanha@redhat.com>
> Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
> Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Message-Id: <20231201093633.2551497-4-alex.bennee@linaro.org>
>
> diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
> index 231b4f68e5..ba2f99b53a 100644
> --- a/tests/avocado/boot_linux_console.py
> +++ b/tests/avocado/boot_linux_console.py
> @@ -191,47 +191,6 @@ def test_mips64el_fuloong2e(self):
> console_pattern = 'Kernel command line: %s' % kernel_command_line
> self.wait_for_console_pattern(console_pattern)
>
> - def test_mips_malta_cpio(self):
> - """
> - :avocado: tags=arch:mips
> - :avocado: tags=machine:malta
> - :avocado: tags=endian:big
> - """
> - deb_url = ('http://snapshot.debian.org/archive/debian/'
> - '20160601T041800Z/pool/main/l/linux/'
> - 'linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb')
In https://lore.kernel.org/qemu-devel/87o7fa5505.fsf@draig.linaro.org/
you mention the linux-4.7 kernel but now you are removing the 4.5 which
also works...
$ wget
http://snapshot.debian.org/archive/debian/20160601T041800Z/pool/main/l/linux/linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb
--2023-12-01 17:46:49--
http://snapshot.debian.org/archive/debian/20160601T041800Z/pool/main/l/linux/linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb
Resolving snapshot.debian.org (snapshot.debian.org)... 185.17.185.185,
193.62.202.27
Connecting to snapshot.debian.org
(snapshot.debian.org)|185.17.185.185|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 29893440 (29M)
Saving to: ‘linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb’
linux-image-4.5.0-2-4kc-malta_4 12%[=> ] 3,52M 1,07MB/s eta 25s
^C
Can we not rush and figure out what is the problem please?
> - deb_hash = 'a3c84f3e88b54e06107d65a410d1d1e8e0f340f8'
> - deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
> - kernel_path = self.extract_from_deb(deb_path,
> - '/boot/vmlinux-4.5.0-2-4kc-malta')
> - initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
> - '8584a59ed9e5eb5ee7ca91f6d74bbb06619205b8/rootfs/'
> - 'mips/rootfs.cpio.gz')
> - initrd_hash = 'bf806e17009360a866bf537f6de66590de349a99'
> - initrd_path_gz = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
> - initrd_path = self.workdir + "rootfs.cpio"
> - archive.gzip_uncompress(initrd_path_gz, initrd_path)
> -
> - self.vm.set_console()
> - kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
> - + 'console=ttyS0 console=tty '
> - + 'rdinit=/sbin/init noreboot')
> - self.vm.add_args('-kernel', kernel_path,
> - '-initrd', initrd_path,
> - '-append', kernel_command_line,
> - '-no-reboot')
> - self.vm.launch()
> - self.wait_for_console_pattern('Boot successful.')
> -
> - exec_command_and_wait_for_pattern(self, 'cat /proc/cpuinfo',
> - 'BogoMIPS')
> - exec_command_and_wait_for_pattern(self, 'uname -a',
> - 'Debian')
> - exec_command_and_wait_for_pattern(self, 'reboot',
> - 'reboot: Restarting system')
> - # Wait for VM to shut down gracefully
> - self.vm.wait()
> -
> @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
> def test_mips64el_malta_5KEc_cpio(self):
> """
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PULL 3/6] tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test
2023-12-01 16:49 ` Philippe Mathieu-Daudé
@ 2023-12-01 17:46 ` Alex Bennée
0 siblings, 0 replies; 12+ messages in thread
From: Alex Bennée @ 2023-12-01 17:46 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: qemu-devel, Stefan Hajnoczi, Cleber Rosa,
Wainer dos Santos Moschetta, Beraldo Leal, Jiaxun Yang
Philippe Mathieu-Daudé <philmd@linaro.org> writes:
> On 1/12/23 16:12, Alex Bennée wrote:
>> The assets are no longer archived by Debian so we can't run this on
>> CI. While some people may still have the test in their cache we do
>> have more recent images from tuxrun so this isn't a great loss.
>> Cc: Stefan Hajnoczi <stefanha@redhat.com>
>> Cc: Philippe Mathieu-Daudé <philmd@linaro.org>
>> Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Message-Id: <20231201093633.2551497-4-alex.bennee@linaro.org>
>> diff --git a/tests/avocado/boot_linux_console.py
>> b/tests/avocado/boot_linux_console.py
>> index 231b4f68e5..ba2f99b53a 100644
>> --- a/tests/avocado/boot_linux_console.py
>> +++ b/tests/avocado/boot_linux_console.py
>> @@ -191,47 +191,6 @@ def test_mips64el_fuloong2e(self):
>> console_pattern = 'Kernel command line: %s' % kernel_command_line
>> self.wait_for_console_pattern(console_pattern)
>> - def test_mips_malta_cpio(self):
>> - """
>> - :avocado: tags=arch:mips
>> - :avocado: tags=machine:malta
>> - :avocado: tags=endian:big
>> - """
>> - deb_url = ('http://snapshot.debian.org/archive/debian/'
>> - '20160601T041800Z/pool/main/l/linux/'
>> - 'linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb')
>
> In https://lore.kernel.org/qemu-devel/87o7fa5505.fsf@draig.linaro.org/
> you mention the linux-4.7 kernel but now you are removing the 4.5
> which
> also works...
>
> $ wget
> http://snapshot.debian.org/archive/debian/20160601T041800Z/pool/main/l/linux/linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb
> --2023-12-01 17:46:49--
> http://snapshot.debian.org/archive/debian/20160601T041800Z/pool/main/l/linux/linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb
> Resolving snapshot.debian.org (snapshot.debian.org)... 185.17.185.185,
> 193.62.202.27
> Connecting to snapshot.debian.org
> (snapshot.debian.org)|185.17.185.185|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 29893440 (29M)
> Saving to: ‘linux-image-4.5.0-2-4kc-malta_4.5.5-1_mips.deb’
>
> linux-image-4.5.0-2-4kc-malta_4 12%[=> ] 3,52M 1,07MB/s eta 25s
> ^C
>
> Can we not rush and figure out what is the problem please?
This was the failure on gitlab:
https://gitlab.com/qemu-project/qemu/-/jobs/5637828476
>
>> - deb_hash = 'a3c84f3e88b54e06107d65a410d1d1e8e0f340f8'
>> - deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
>> - kernel_path = self.extract_from_deb(deb_path,
>> - '/boot/vmlinux-4.5.0-2-4kc-malta')
>> - initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
>> - '8584a59ed9e5eb5ee7ca91f6d74bbb06619205b8/rootfs/'
>> - 'mips/rootfs.cpio.gz')
>> - initrd_hash = 'bf806e17009360a866bf537f6de66590de349a99'
>> - initrd_path_gz = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
>> - initrd_path = self.workdir + "rootfs.cpio"
>> - archive.gzip_uncompress(initrd_path_gz, initrd_path)
>> -
>> - self.vm.set_console()
>> - kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE
>> - + 'console=ttyS0 console=tty '
>> - + 'rdinit=/sbin/init noreboot')
>> - self.vm.add_args('-kernel', kernel_path,
>> - '-initrd', initrd_path,
>> - '-append', kernel_command_line,
>> - '-no-reboot')
>> - self.vm.launch()
>> - self.wait_for_console_pattern('Boot successful.')
>> -
>> - exec_command_and_wait_for_pattern(self, 'cat /proc/cpuinfo',
>> - 'BogoMIPS')
>> - exec_command_and_wait_for_pattern(self, 'uname -a',
>> - 'Debian')
>> - exec_command_and_wait_for_pattern(self, 'reboot',
>> - 'reboot: Restarting system')
>> - # Wait for VM to shut down gracefully
>> - self.vm.wait()
>> -
>> @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
>> def test_mips64el_malta_5KEc_cpio(self):
>> """
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PULL 4/6] tests/avocado: tag sbsa tests as tcg only
2023-12-01 15:12 [PULL 0/6] gdbstub, avocado and gitlab updates Alex Bennée
` (2 preceding siblings ...)
2023-12-01 15:12 ` [PULL 3/6] tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test Alex Bennée
@ 2023-12-01 15:12 ` Alex Bennée
2023-12-01 15:12 ` [PULL 5/6] gitlab: build the correct microblaze target Alex Bennée
` (2 subsequent siblings)
6 siblings, 0 replies; 12+ messages in thread
From: Alex Bennée @ 2023-12-01 15:12 UTC (permalink / raw)
To: qemu-devel
Cc: Alex Bennée, Philippe Mathieu-Daudé, Stefan Hajnoczi,
Radoslaw Biernacki, Peter Maydell, Leif Lindholm,
Marcin Juszkiewicz, Cleber Rosa, Wainer dos Santos Moschetta,
Beraldo Leal, open list:SBSA-REF
As firmware runs at a higher privilege level than the hypervisor we
can only run these tests under TCG emulation.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231201093633.2551497-6-alex.bennee@linaro.org>
diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
index bdd1efc768..528c7d2934 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -20,6 +20,10 @@ class Aarch64SbsarefMachine(QemuSystemTest):
"""
:avocado: tags=arch:aarch64
:avocado: tags=machine:sbsa-ref
+ :avocado: tags=accel:tcg
+
+ As firmware runs at a higher privilege level than the hypervisor we
+ can only run these tests under TCG emulation.
"""
timeout = 180
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PULL 5/6] gitlab: build the correct microblaze target
2023-12-01 15:12 [PULL 0/6] gdbstub, avocado and gitlab updates Alex Bennée
` (3 preceding siblings ...)
2023-12-01 15:12 ` [PULL 4/6] tests/avocado: tag sbsa tests as tcg only Alex Bennée
@ 2023-12-01 15:12 ` Alex Bennée
2023-12-01 15:12 ` [PULL 6/6] gitlab: add optional job to run flaky avocado tests Alex Bennée
2023-12-01 16:50 ` [PULL 0/6] gdbstub, avocado and gitlab updates Philippe Mathieu-Daudé
6 siblings, 0 replies; 12+ messages in thread
From: Alex Bennée @ 2023-12-01 15:12 UTC (permalink / raw)
To: qemu-devel
Cc: Alex Bennée, Philippe Mathieu-Daudé, Stefan Hajnoczi,
Thomas Huth, Wainer dos Santos Moschetta, Beraldo Leal
We inadvertently built the LE target for BE tests.
Fixes: 78ebc00b06 (gitlab: shuffle some targets and reduce avocado noise)
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231201093633.2551497-7-alex.bennee@linaro.org>
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 7f9af83b10..62b5379a5e 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -41,7 +41,7 @@ build-system-ubuntu:
variables:
IMAGE: ubuntu2204
CONFIGURE_ARGS: --enable-docs
- TARGETS: alpha-softmmu microblazeel-softmmu mips64el-softmmu
+ TARGETS: alpha-softmmu microblaze-softmmu mips64el-softmmu
MAKE_CHECK_ARGS: check-build
check-system-ubuntu:
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PULL 6/6] gitlab: add optional job to run flaky avocado tests
2023-12-01 15:12 [PULL 0/6] gdbstub, avocado and gitlab updates Alex Bennée
` (4 preceding siblings ...)
2023-12-01 15:12 ` [PULL 5/6] gitlab: build the correct microblaze target Alex Bennée
@ 2023-12-01 15:12 ` Alex Bennée
2023-12-01 16:50 ` [PULL 0/6] gdbstub, avocado and gitlab updates Philippe Mathieu-Daudé
6 siblings, 0 replies; 12+ messages in thread
From: Alex Bennée @ 2023-12-01 15:12 UTC (permalink / raw)
To: qemu-devel
Cc: Alex Bennée, Richard Henderson, Stefan Hajnoczi,
Philippe Mathieu-Daudé, Thomas Huth,
Wainer dos Santos Moschetta, Beraldo Leal, Cleber Rosa,
Cédric Le Goater, Peter Maydell, Andrew Jeffery,
Joel Stanley, Aurelien Jarno, Yoshinori Sato, Halil Pasic,
Christian Borntraeger, Eric Farman, Pavel Dovgalyuk,
Paolo Bonzini, Eric Auger, open list:ASPEED BMCs,
open list:S390 Virtio-ccw
One problem with flaky tests is they often only fail under CI
conditions which makes it hard to debug. We add an optional allow_fail
job so developers can trigger the only the flaky tests in the CI
environment if they are debugging.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231201093633.2551497-8-alex.bennee@linaro.org>
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 76465b8f3d..bd132306c1 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -1387,16 +1387,17 @@ the code snippet below:
def test(self):
do_something()
-Tests should not live in this state forever and should either be fixed
-or eventually removed.
-
-To run such tests locally you will need to set the environment
-variable. For example:
+You can also add ``:avocado: tags=flaky`` to the test meta-data so
+only the flaky tests can be run as a group:
.. code::
- env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado run \
- tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg
+ env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado \
+ run tests/avocado -filter-by-tags=flaky
+
+Tests should not live in this state forever and should either be fixed
+or eventually removed.
+
Uninstalling Avocado
~~~~~~~~~~~~~~~~~~~~
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 62b5379a5e..91663946de 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -217,6 +217,36 @@ avocado-system-opensuse:
MAKE_CHECK_ARGS: check-avocado
AVOCADO_TAGS: arch:s390x arch:x86_64 arch:aarch64
+#
+# Flaky tests. We don't run these by default and they are allow fail
+# but often the CI system is the only way to trigger the failures.
+#
+
+build-system-flaky:
+ extends:
+ - .native_build_job_template
+ - .native_build_artifact_template
+ needs:
+ job: amd64-debian-container
+ variables:
+ IMAGE: debian
+ QEMU_JOB_OPTIONAL: 1
+ TARGETS: aarch64-softmmu arm-softmmu mips64el-softmmu
+ ppc64-softmmu rx-softmmu s390x-softmmu sh4-softmmu x86_64-softmmu
+ MAKE_CHECK_ARGS: check-build
+
+avocado-system-flaky:
+ extends: .avocado_test_job_template
+ needs:
+ - job: build-system-flaky
+ artifacts: true
+ allow_failure: true
+ variables:
+ IMAGE: debian
+ MAKE_CHECK_ARGS: check-avocado
+ QEMU_JOB_OPTIONAL: 1
+ QEMU_TEST_FLAKY_TESTS: 1
+ AVOCADO_TAGS: flaky
# This jobs explicitly disable TCG (--disable-tcg), KVM is detected by
# the configure script. The container doesn't contain Xen headers so
diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py
index 9e9773e6e1..7c4769904e 100644
--- a/tests/avocado/boot_linux.py
+++ b/tests/avocado/boot_linux.py
@@ -99,6 +99,7 @@ def test_pseries_tcg(self):
"""
:avocado: tags=machine:pseries
:avocado: tags=accel:tcg
+ :avocado: tags=flaky
"""
self.require_accelerator("tcg")
self.vm.add_args("-accel", "tcg")
@@ -118,6 +119,7 @@ def test_s390_ccw_virtio_tcg(self):
"""
:avocado: tags=machine:s390-ccw-virtio
:avocado: tags=accel:tcg
+ :avocado: tags=flaky
"""
self.require_accelerator("tcg")
self.vm.add_args("-accel", "tcg")
diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
index ba2f99b53a..7487337f46 100644
--- a/tests/avocado/boot_linux_console.py
+++ b/tests/avocado/boot_linux_console.py
@@ -1384,6 +1384,7 @@ def test_sh4_r2d(self):
"""
:avocado: tags=arch:sh4
:avocado: tags=machine:r2d
+ :avocado: tags=flaky
"""
tar_hash = 'fe06a4fd8ccbf2e27928d64472939d47829d4c7e'
self.vm.add_args('-append', 'console=ttySC1')
diff --git a/tests/avocado/intel_iommu.py b/tests/avocado/intel_iommu.py
index 2dd11a6346..f04ee1cf9d 100644
--- a/tests/avocado/intel_iommu.py
+++ b/tests/avocado/intel_iommu.py
@@ -22,6 +22,7 @@ class IntelIOMMU(LinuxTest):
:avocado: tags=machine:q35
:avocado: tags=accel:kvm
:avocado: tags=intel_iommu
+ :avocado: tags=flaky
"""
IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'
diff --git a/tests/avocado/linux_initrd.py b/tests/avocado/linux_initrd.py
index c40a987bd1..aad5b19bd9 100644
--- a/tests/avocado/linux_initrd.py
+++ b/tests/avocado/linux_initrd.py
@@ -57,6 +57,8 @@ def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self):
def test_with_2gib_file_should_work_with_linux_v4_16(self):
"""
+ :avocado: tags=flaky
+
QEMU has supported up to 4 GiB initrd for recent kernel
Expect guest can reach 'Unpacking initramfs...'
"""
diff --git a/tests/avocado/machine_aspeed.py b/tests/avocado/machine_aspeed.py
index 258fb50c47..6fa5459a07 100644
--- a/tests/avocado/machine_aspeed.py
+++ b/tests/avocado/machine_aspeed.py
@@ -317,6 +317,7 @@ def test_arm_ast2500_evb_sdk(self):
"""
:avocado: tags=arch:arm
:avocado: tags=machine:ast2500-evb
+ :avocado: tags=flaky
"""
image_url = ('https://github.com/AspeedTech-BMC/openbmc/releases/'
@@ -336,6 +337,7 @@ def test_arm_ast2600_evb_sdk(self):
"""
:avocado: tags=arch:arm
:avocado: tags=machine:ast2600-evb
+ :avocado: tags=flaky
"""
image_url = ('https://github.com/AspeedTech-BMC/openbmc/releases/'
diff --git a/tests/avocado/machine_mips_malta.py b/tests/avocado/machine_mips_malta.py
index 959dcf5602..99bee49e9a 100644
--- a/tests/avocado/machine_mips_malta.py
+++ b/tests/avocado/machine_mips_malta.py
@@ -109,6 +109,7 @@ def test_mips_malta_i6400_framebuffer_logo_7cores(self):
:avocado: tags=machine:malta
:avocado: tags=cpu:I6400
:avocado: tags=mips:smp
+ :avocado: tags=flaky
"""
self.do_test_i6400_framebuffer_logo(7)
@@ -120,6 +121,7 @@ def test_mips_malta_i6400_framebuffer_logo_8cores(self):
:avocado: tags=machine:malta
:avocado: tags=cpu:I6400
:avocado: tags=mips:smp
+ :avocado: tags=flaky
"""
self.do_test_i6400_framebuffer_logo(8)
diff --git a/tests/avocado/machine_rx_gdbsim.py b/tests/avocado/machine_rx_gdbsim.py
index 350a73fbbb..412a7a5089 100644
--- a/tests/avocado/machine_rx_gdbsim.py
+++ b/tests/avocado/machine_rx_gdbsim.py
@@ -31,6 +31,7 @@ def test_uboot(self):
:avocado: tags=arch:rx
:avocado: tags=machine:gdbsim-r5f562n8
:avocado: tags=endian:little
+ :avocado: tags=flaky
"""
uboot_url = ('https://acc.dl.osdn.jp/users/23/23888/u-boot.bin.gz')
uboot_hash = '9b78dbd43b40b2526848c0b1ce9de02c24f4dcdb'
@@ -56,6 +57,7 @@ def test_linux_sash(self):
:avocado: tags=arch:rx
:avocado: tags=machine:gdbsim-r5f562n7
:avocado: tags=endian:little
+ :avocado: tags=flaky
"""
dtb_url = ('https://acc.dl.osdn.jp/users/23/23887/rx-virt.dtb')
dtb_hash = '7b4e4e2c71905da44e86ce47adee2210b026ac18'
diff --git a/tests/avocado/machine_s390_ccw_virtio.py b/tests/avocado/machine_s390_ccw_virtio.py
index 61e75d8f9e..26e938c9e9 100644
--- a/tests/avocado/machine_s390_ccw_virtio.py
+++ b/tests/avocado/machine_s390_ccw_virtio.py
@@ -167,6 +167,7 @@ def test_s390x_fedora(self):
:avocado: tags=device:virtio-gpu
:avocado: tags=device:virtio-crypto
:avocado: tags=device:virtio-net
+ :avocado: tags=flaky
"""
kernel_url = ('https://archives.fedoraproject.org/pub/archive'
diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py
index 0d32cc280e..af086eab08 100644
--- a/tests/avocado/replay_kernel.py
+++ b/tests/avocado/replay_kernel.py
@@ -88,6 +88,7 @@ def test_x86_64_pc(self):
"""
:avocado: tags=arch:x86_64
:avocado: tags=machine:pc
+ :avocado: tags=flaky
"""
kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora'
'/linux/releases/29/Everything/x86_64/os/images/pxeboot'
@@ -186,6 +187,7 @@ def test_arm_cubieboard_initrd(self):
"""
:avocado: tags=arch:arm
:avocado: tags=machine:cubieboard
+ :avocado: tags=flaky
"""
deb_url = ('https://apt.armbian.com/pool/main/l/'
'linux-5.10.16-sunxi/linux-image-current-sunxi_21.02.2_armhf.deb')
diff --git a/tests/avocado/reverse_debugging.py b/tests/avocado/reverse_debugging.py
index 9a468321e5..4cce5a5598 100644
--- a/tests/avocado/reverse_debugging.py
+++ b/tests/avocado/reverse_debugging.py
@@ -255,6 +255,7 @@ def test_ppc64_pseries(self):
"""
:avocado: tags=arch:ppc64
:avocado: tags=machine:pseries
+ :avocado: tags=flaky
"""
# SLOF branches back to its entry point, which causes this test
# to take the 'hit a breakpoint again' path. That's not a problem,
@@ -269,6 +270,7 @@ def test_ppc64_powernv(self):
"""
:avocado: tags=arch:ppc64
:avocado: tags=machine:powernv
+ :avocado: tags=flaky
"""
self.endian_is_le = False
self.reverse_debugging()
diff --git a/tests/avocado/smmu.py b/tests/avocado/smmu.py
index 05b34418a5..21ff030ca7 100644
--- a/tests/avocado/smmu.py
+++ b/tests/avocado/smmu.py
@@ -22,6 +22,7 @@ class SMMU(LinuxTest):
:avocado: tags=machine:virt
:avocado: tags=distro:fedora
:avocado: tags=smmu
+ :avocado: tags=flaky
"""
IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'
diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py
index 5f859f4e6f..a936a3b780 100644
--- a/tests/avocado/tuxrun_baselines.py
+++ b/tests/avocado/tuxrun_baselines.py
@@ -561,6 +561,7 @@ def test_sh4(self):
:avocado: tags=image:zImage
:avocado: tags=root:sda
:avocado: tags=console:ttySC1
+ :avocado: tags=flaky
"""
sums = { "rootfs.ext4.zst" :
"3592a7a3d5a641e8b9821449e77bc43c9904a56c30d45da0694349cfd86743fd",
--
2.39.2
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PULL 0/6] gdbstub, avocado and gitlab updates
2023-12-01 15:12 [PULL 0/6] gdbstub, avocado and gitlab updates Alex Bennée
` (5 preceding siblings ...)
2023-12-01 15:12 ` [PULL 6/6] gitlab: add optional job to run flaky avocado tests Alex Bennée
@ 2023-12-01 16:50 ` Philippe Mathieu-Daudé
2023-12-01 16:52 ` Philippe Mathieu-Daudé
6 siblings, 1 reply; 12+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-12-01 16:50 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
Hi Alex,
On 1/12/23 16:12, Alex Bennée wrote:
> The following changes since commit abf635ddfe3242df907f58967f3c1e6763bbca2d:
>
> Update version for v8.2.0-rc2 release (2023-11-28 16:31:16 -0500)
>
> are available in the Git repository at:
>
> https://gitlab.com/stsquad/qemu.git tags/pull-more-8.2-fixes-011223-1
>
> for you to fetch changes up to a7a740cce1ec49568e7ebd8e776279e59e2fb5b6:
>
> gitlab: add optional job to run flaky avocado tests (2023-12-01 14:12:05 +0000)
>
> ----------------------------------------------------------------
> Flaky avocado tests, gdbstub and gitlab tweaks
>
> - gdbstub, properly halt when QEMU is having IO issues
> - convert skipIf(GITLAB_CI) to skipUnless(QEMU_TEST_FLAKY_TESTS)
> - drop mips malta cpio test as assets expire in Debian archive
> - tag sbsa-ref tests as TCG only
> - build the correct microblaze for avocado-system-ubuntu
> - add optional flaky tests job to CI
>
> ----------------------------------------------------------------
> Alex Bennée (6):
> gdbstub: use a better signal when we halt for IO reasons
> docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS
> tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test
This patch has been posted 5h ago and we are still discussing it...
I'm quite surprised you include it in your PR.
> tests/avocado: tag sbsa tests as tcg only
> gitlab: build the correct microblaze target
> gitlab: add optional job to run flaky avocado tests
>
> docs/devel/testing.rst | 32 ++++++++++++++--------
> gdbstub/internals.h | 1 +
> gdbstub/system.c | 2 +-
> .gitlab-ci.d/buildtest.yml | 32 +++++++++++++++++++++-
> tests/avocado/boot_linux.py | 10 +++++--
> tests/avocado/boot_linux_console.py | 47 +++-----------------------------
> tests/avocado/intel_iommu.py | 6 ++--
> tests/avocado/linux_initrd.py | 7 +++--
> tests/avocado/machine_aarch64_sbsaref.py | 4 +++
> tests/avocado/machine_aspeed.py | 10 +++++--
> tests/avocado/machine_mips_malta.py | 10 +++++--
> tests/avocado/machine_rx_gdbsim.py | 10 +++++--
> tests/avocado/machine_s390_ccw_virtio.py | 3 +-
> tests/avocado/replay_kernel.py | 8 ++++--
> tests/avocado/reverse_debugging.py | 16 +++++++----
> tests/avocado/smmu.py | 6 ++--
> tests/avocado/tuxrun_baselines.py | 5 ++--
> 17 files changed, 125 insertions(+), 84 deletions(-)
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PULL 0/6] gdbstub, avocado and gitlab updates
2023-12-01 16:50 ` [PULL 0/6] gdbstub, avocado and gitlab updates Philippe Mathieu-Daudé
@ 2023-12-01 16:52 ` Philippe Mathieu-Daudé
2023-12-01 17:50 ` Alex Bennée
0 siblings, 1 reply; 12+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-12-01 16:52 UTC (permalink / raw)
To: Alex Bennée, qemu-devel
On 1/12/23 17:50, Philippe Mathieu-Daudé wrote:
> Hi Alex,
>
> On 1/12/23 16:12, Alex Bennée wrote:
>> The following changes since commit
>> abf635ddfe3242df907f58967f3c1e6763bbca2d:
>>
>> Update version for v8.2.0-rc2 release (2023-11-28 16:31:16 -0500)
>>
>> are available in the Git repository at:
>>
>> https://gitlab.com/stsquad/qemu.git tags/pull-more-8.2-fixes-011223-1
>>
>> for you to fetch changes up to a7a740cce1ec49568e7ebd8e776279e59e2fb5b6:
>>
>> gitlab: add optional job to run flaky avocado tests (2023-12-01
>> 14:12:05 +0000)
>>
>> ----------------------------------------------------------------
>> Flaky avocado tests, gdbstub and gitlab tweaks
>>
>> - gdbstub, properly halt when QEMU is having IO issues
>> - convert skipIf(GITLAB_CI) to skipUnless(QEMU_TEST_FLAKY_TESTS)
>> - drop mips malta cpio test as assets expire in Debian archive
>> - tag sbsa-ref tests as TCG only
>> - build the correct microblaze for avocado-system-ubuntu
>> - add optional flaky tests job to CI
>>
>> ----------------------------------------------------------------
>> Alex Bennée (6):
>> gdbstub: use a better signal when we halt for IO reasons
>> docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS
>> tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test
>
> This patch has been posted 5h ago and we are still discussing it...
> I'm quite surprised you include it in your PR.
Do you mind re-posting your PR without this patch? I don't
mind to work on it during the WE and merge it let's say
Sunday, if you want it in before this Monday.
Regards,
Phil.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PULL 0/6] gdbstub, avocado and gitlab updates
2023-12-01 16:52 ` Philippe Mathieu-Daudé
@ 2023-12-01 17:50 ` Alex Bennée
0 siblings, 0 replies; 12+ messages in thread
From: Alex Bennée @ 2023-12-01 17:50 UTC (permalink / raw)
To: Philippe Mathieu-Daudé; +Cc: qemu-devel
Philippe Mathieu-Daudé <philmd@linaro.org> writes:
> On 1/12/23 17:50, Philippe Mathieu-Daudé wrote:
>> Hi Alex,
>> On 1/12/23 16:12, Alex Bennée wrote:
>>> The following changes since commit
>>> abf635ddfe3242df907f58967f3c1e6763bbca2d:
>>>
>>> Update version for v8.2.0-rc2 release (2023-11-28 16:31:16 -0500)
>>>
>>> are available in the Git repository at:
>>>
>>> https://gitlab.com/stsquad/qemu.git tags/pull-more-8.2-fixes-011223-1
>>>
>>> for you to fetch changes up to a7a740cce1ec49568e7ebd8e776279e59e2fb5b6:
>>>
>>> gitlab: add optional job to run flaky avocado tests (2023-12-01
>>> 14:12:05 +0000)
>>>
>>> ----------------------------------------------------------------
>>> Flaky avocado tests, gdbstub and gitlab tweaks
>>>
>>> - gdbstub, properly halt when QEMU is having IO issues
>>> - convert skipIf(GITLAB_CI) to skipUnless(QEMU_TEST_FLAKY_TESTS)
>>> - drop mips malta cpio test as assets expire in Debian archive
>>> - tag sbsa-ref tests as TCG only
>>> - build the correct microblaze for avocado-system-ubuntu
>>> - add optional flaky tests job to CI
>>>
>>> ----------------------------------------------------------------
>>> Alex Bennée (6):
>>> gdbstub: use a better signal when we halt for IO reasons
>>> docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS
>>> tests/avocado: drop BootLinuxConsole.test_mips_malta_cpio test
>> This patch has been posted 5h ago and we are still discussing it...
>> I'm quite surprised you include it in your PR.
>
> Do you mind re-posting your PR without this patch? I don't
> mind to work on it during the WE and merge it let's say
> Sunday, if you want it in before this Monday.
I've sent v2 of the PR without the mips changes.
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 12+ messages in thread