* [PATCH 0/3] final fixes for 8.2
@ 2023-11-30 15:33 Alex Bennée
2023-11-30 15:33 ` [PATCH 1/3] gdbstub: use a better signal when we halt for IO reasons Alex Bennée
` (3 more replies)
0 siblings, 4 replies; 9+ messages in thread
From: Alex Bennée @ 2023-11-30 15:33 UTC (permalink / raw)
To: qemu-devel
Cc: Eric Farman, Cleber Rosa, Andrew Jeffery,
Wainer dos Santos Moschetta, Philippe Mathieu-Daudé,
Yoshinori Sato, Cédric Le Goater, Alex Bennée,
Eric Auger, qemu-arm, Peter Maydell, Halil Pasic, Pavel Dovgalyuk,
Aurelien Jarno, qemu-s390x, Beraldo Leal, Joel Stanley,
Paolo Bonzini, Christian Borntraeger, Thomas Huth
8.2 is looking fairly stable but I do have one bug fix for gdbstub
which I came across while debugging something else. The changes for
avocado rationalise all flaky tests under the QEMU_TEST_FLAKY_TESTS
environment variable. The final patch re-adds the flaky tests to the
CI as a manually run allow_fail job so we can still attempt to debug
their failure in the place they tend to fall over.
Alex.
Alex Bennée (3):
gdbstub: use a better signal when we halt for IO reasons
docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS
gitlab: add optional job to run flaky avocado tests
docs/devel/testing.rst | 32 ++++++++++++++++--------
gdbstub/internals.h | 1 +
gdbstub/system.c | 2 +-
.gitlab-ci.d/buildtest.yml | 30 ++++++++++++++++++++++
tests/avocado/boot_linux.py | 10 +++++---
tests/avocado/boot_linux_console.py | 6 +++--
tests/avocado/intel_iommu.py | 6 +++--
tests/avocado/linux_initrd.py | 7 ++++--
tests/avocado/machine_aspeed.py | 10 +++++---
tests/avocado/machine_mips_malta.py | 10 +++++---
tests/avocado/machine_rx_gdbsim.py | 10 +++++---
tests/avocado/machine_s390_ccw_virtio.py | 3 ++-
tests/avocado/replay_kernel.py | 7 ++++--
tests/avocado/reverse_debugging.py | 16 ++++++++----
tests/avocado/smmu.py | 6 +++--
tests/avocado/tuxrun_baselines.py | 5 ++--
16 files changed, 119 insertions(+), 42 deletions(-)
--
2.39.2
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH 1/3] gdbstub: use a better signal when we halt for IO reasons 2023-11-30 15:33 [PATCH 0/3] final fixes for 8.2 Alex Bennée @ 2023-11-30 15:33 ` Alex Bennée 2023-11-30 15:33 ` [PATCH 2/3] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS Alex Bennée ` (2 subsequent siblings) 3 siblings, 0 replies; 9+ messages in thread From: Alex Bennée @ 2023-11-30 15:33 UTC (permalink / raw) To: qemu-devel Cc: Eric Farman, Cleber Rosa, Andrew Jeffery, Wainer dos Santos Moschetta, Philippe Mathieu-Daudé, Yoshinori Sato, Cédric Le Goater, Alex Bennée, Eric Auger, qemu-arm, Peter Maydell, Halil Pasic, Pavel Dovgalyuk, Aurelien Jarno, qemu-s390x, Beraldo Leal, Joel Stanley, Paolo Bonzini, Christian Borntraeger, Thomas Huth, Luis Machado The gdb description GDB_SIGNAL_IO is "I/O possible" and by default gdb will try and restart the guest, getting us nowhere. Report GDB_SIGNAL_STOP instead which should at least halt the session at the failure point. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Cc: Luis Machado <luis.machado@arm.com> Message-Id: <20231123131905.2640498-1-alex.bennee@linaro.org> --- gdbstub/internals.h | 1 + gdbstub/system.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/gdbstub/internals.h b/gdbstub/internals.h index 465c24b36e..5c0c725e54 100644 --- a/gdbstub/internals.h +++ b/gdbstub/internals.h @@ -24,6 +24,7 @@ enum { GDB_SIGNAL_TRAP = 5, GDB_SIGNAL_ABRT = 6, GDB_SIGNAL_ALRM = 14, + GDB_SIGNAL_STOP = 17, GDB_SIGNAL_IO = 23, GDB_SIGNAL_XCPU = 24, GDB_SIGNAL_UNKNOWN = 143 diff --git a/gdbstub/system.c b/gdbstub/system.c index 783ac140b9..83fd452800 100644 --- a/gdbstub/system.c +++ b/gdbstub/system.c @@ -183,7 +183,7 @@ static void gdb_vm_state_change(void *opaque, bool running, RunState state) break; case RUN_STATE_IO_ERROR: trace_gdbstub_hit_io_error(); - ret = GDB_SIGNAL_IO; + ret = GDB_SIGNAL_STOP; break; case RUN_STATE_WATCHDOG: trace_gdbstub_hit_watchdog(); -- 2.39.2 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 2/3] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS 2023-11-30 15:33 [PATCH 0/3] final fixes for 8.2 Alex Bennée 2023-11-30 15:33 ` [PATCH 1/3] gdbstub: use a better signal when we halt for IO reasons Alex Bennée @ 2023-11-30 15:33 ` Alex Bennée 2023-11-30 16:06 ` Cédric Le Goater ` (2 more replies) 2023-11-30 15:33 ` [PATCH 3/3] gitlab: add optional job to run flaky avocado tests Alex Bennée 2023-11-30 16:05 ` [PATCH 0/3] final fixes for 8.2 Richard Henderson 3 siblings, 3 replies; 9+ messages in thread From: Alex Bennée @ 2023-11-30 15:33 UTC (permalink / raw) To: qemu-devel Cc: Eric Farman, Cleber Rosa, Andrew Jeffery, Wainer dos Santos Moschetta, Philippe Mathieu-Daudé, Yoshinori Sato, Cédric Le Goater, Alex Bennée, Eric Auger, qemu-arm, Peter Maydell, Halil Pasic, Pavel Dovgalyuk, Aurelien Jarno, qemu-s390x, Beraldo Leal, Joel Stanley, Paolo Bonzini, Christian Borntraeger, Thomas Huth It doesn't make sense to have two classes of flaky tests. While it may take the constrained environment of CI to trigger failures easily it doesn't mean they don't occasionally happen on developer machines. As CI is the gating factor to passing there is no point developers running the tests locally anyway unless they are trying to fix things. While we are at it update the language in the docs to discourage the QEMU_TEST_FLAKY_TESTS becoming a permanent solution. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> --- docs/devel/testing.rst | 31 +++++++++++++++--------- tests/avocado/boot_linux.py | 8 +++--- tests/avocado/boot_linux_console.py | 5 ++-- tests/avocado/intel_iommu.py | 5 ++-- tests/avocado/linux_initrd.py | 5 ++-- tests/avocado/machine_aspeed.py | 8 +++--- tests/avocado/machine_mips_malta.py | 8 +++--- tests/avocado/machine_rx_gdbsim.py | 8 +++--- tests/avocado/machine_s390_ccw_virtio.py | 2 +- tests/avocado/replay_kernel.py | 5 ++-- tests/avocado/reverse_debugging.py | 14 +++++++---- tests/avocado/smmu.py | 5 ++-- tests/avocado/tuxrun_baselines.py | 4 +-- 13 files changed, 67 insertions(+), 41 deletions(-) diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 22218dbedb..579d1837e0 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -1371,23 +1371,32 @@ conditions. For example, tests that take longer to execute when QEMU is compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable has been used to determine whether those tests should run or not. -GITLAB_CI -^^^^^^^^^ -A number of tests are flagged to not run on the GitLab CI. Usually because -they proved to the flaky or there are constraints on the CI environment which -would make them fail. If you encounter a similar situation then use that -variable as shown on the code snippet below to skip the test: +QEMU_TEST_FLAKY_TESTS +^^^^^^^^^^^^^^^^^^^^^ +Some tests are not working reliably and thus are disabled by default. +This includes tests that don't run reliably on GitLab's CI which +usually expose real issues that are rarely seen on developer machines +due to the constraints of the CI environment. If you encounter a +similar situation then mark the test as shown on the code snippet +below: .. code:: - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') def test(self): do_something() -QEMU_TEST_FLAKY_TESTS -^^^^^^^^^^^^^^^^^^^^^ -Some tests are not working reliably and thus are disabled by default. -Set this environment variable to enable them. +Tests should not live in this state forever and should either be fixed +or eventually removed. If you move a test into this category please +consider raising a bug to track progress towards a fix. + +To run such tests locally you will need to set the environment +variable. For example: + +.. code:: + + env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado run \ + tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg Uninstalling Avocado ~~~~~~~~~~~~~~~~~~~~ diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py index be30dcbd58..9e9773e6e1 100644 --- a/tests/avocado/boot_linux.py +++ b/tests/avocado/boot_linux.py @@ -12,7 +12,7 @@ from avocado_qemu import LinuxTest, BUILD_DIR -from avocado import skipIf +from avocado import skipUnless class BootLinuxX8664(LinuxTest): @@ -93,7 +93,8 @@ class BootLinuxPPC64(LinuxTest): timeout = 360 - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_pseries_tcg(self): """ :avocado: tags=machine:pseries @@ -111,7 +112,8 @@ class BootLinuxS390X(LinuxTest): timeout = 240 - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_s390_ccw_virtio_tcg(self): """ :avocado: tags=machine:s390-ccw-virtio diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py index 6eab515718..231b4f68e5 100644 --- a/tests/avocado/boot_linux_console.py +++ b/tests/avocado/boot_linux_console.py @@ -15,7 +15,7 @@ from avocado import skip from avocado import skipUnless -from avocado import skipIf +from avocado import skipUnless from avocado_qemu import QemuSystemTest from avocado_qemu import exec_command from avocado_qemu import exec_command_and_wait_for_pattern @@ -1419,7 +1419,8 @@ def test_ppc_mac99(self): # This test has a 6-10% failure rate on various hosts that look # like issues with a buggy kernel. As a result we don't want it # gating releases on Gitlab. - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_sh4_r2d(self): """ :avocado: tags=arch:sh4 diff --git a/tests/avocado/intel_iommu.py b/tests/avocado/intel_iommu.py index 77635ab56c..2dd11a6346 100644 --- a/tests/avocado/intel_iommu.py +++ b/tests/avocado/intel_iommu.py @@ -9,10 +9,11 @@ # later. See the COPYING file in the top-level directory. import os -from avocado import skipIf +from avocado import skipUnless from avocado_qemu import LinuxTest -@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') +@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + class IntelIOMMU(LinuxTest): """ :avocado: tags=arch:x86_64 diff --git a/tests/avocado/linux_initrd.py b/tests/avocado/linux_initrd.py index ba02e5a563..c40a987bd1 100644 --- a/tests/avocado/linux_initrd.py +++ b/tests/avocado/linux_initrd.py @@ -13,7 +13,7 @@ import tempfile from avocado_qemu import QemuSystemTest -from avocado import skipIf +from avocado import skipUnless class LinuxInitrd(QemuSystemTest): @@ -53,7 +53,8 @@ def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self): max_size + 1) self.assertRegex(self.vm.get_log(), expected_msg) - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_with_2gib_file_should_work_with_linux_v4_16(self): """ QEMU has supported up to 4 GiB initrd for recent kernel diff --git a/tests/avocado/machine_aspeed.py b/tests/avocado/machine_aspeed.py index df31b2a8a8..258fb50c47 100644 --- a/tests/avocado/machine_aspeed.py +++ b/tests/avocado/machine_aspeed.py @@ -18,7 +18,7 @@ from avocado_qemu import interrupt_interactive_console_until_pattern from avocado_qemu import has_cmd from avocado.utils import archive -from avocado import skipIf +from avocado import skipUnless from avocado import skipUnless @@ -311,7 +311,8 @@ def do_test_arm_aspeed_sdk_start(self, image): self, 'boot', '## Loading kernel from FIT Image') self.wait_for_console_pattern('Starting kernel ...') - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_arm_ast2500_evb_sdk(self): """ :avocado: tags=arch:arm @@ -329,7 +330,8 @@ def test_arm_ast2500_evb_sdk(self): self.workdir + '/ast2500-default/image-bmc') self.wait_for_console_pattern('nodistro.0 ast2500-default ttyS4') - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_arm_ast2600_evb_sdk(self): """ :avocado: tags=arch:arm diff --git a/tests/avocado/machine_mips_malta.py b/tests/avocado/machine_mips_malta.py index 9bd54518bf..959dcf5602 100644 --- a/tests/avocado/machine_mips_malta.py +++ b/tests/avocado/machine_mips_malta.py @@ -11,7 +11,7 @@ import gzip import logging -from avocado import skipIf +from avocado import skipUnless from avocado import skipUnless from avocado.utils import archive from avocado_qemu import QemuSystemTest @@ -101,7 +101,8 @@ def test_mips_malta_i6400_framebuffer_logo_1core(self): """ self.do_test_i6400_framebuffer_logo(1) - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_mips_malta_i6400_framebuffer_logo_7cores(self): """ :avocado: tags=arch:mips64el @@ -111,7 +112,8 @@ def test_mips_malta_i6400_framebuffer_logo_7cores(self): """ self.do_test_i6400_framebuffer_logo(7) - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_mips_malta_i6400_framebuffer_logo_8cores(self): """ :avocado: tags=arch:mips64el diff --git a/tests/avocado/machine_rx_gdbsim.py b/tests/avocado/machine_rx_gdbsim.py index 6cd8704b01..350a73fbbb 100644 --- a/tests/avocado/machine_rx_gdbsim.py +++ b/tests/avocado/machine_rx_gdbsim.py @@ -10,7 +10,7 @@ import os -from avocado import skipIf +from avocado import skipUnless from avocado_qemu import QemuSystemTest from avocado_qemu import exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern @@ -22,7 +22,8 @@ class RxGdbSimMachine(QemuSystemTest): timeout = 30 KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 ' - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_uboot(self): """ U-Boot and checks that the console is operational. @@ -46,7 +47,8 @@ def test_uboot(self): # FIXME limit baudrate on chardev, else we type too fast #exec_command_and_wait_for_pattern(self, 'version', gcc_version) - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_linux_sash(self): """ Boots a Linux kernel and checks that the console is operational. diff --git a/tests/avocado/machine_s390_ccw_virtio.py b/tests/avocado/machine_s390_ccw_virtio.py index ffd914ded9..61e75d8f9e 100644 --- a/tests/avocado/machine_s390_ccw_virtio.py +++ b/tests/avocado/machine_s390_ccw_virtio.py @@ -12,7 +12,7 @@ import os import tempfile -from avocado import skipIf +from avocado import skipUnless from avocado_qemu import QemuSystemTest from avocado_qemu import exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py index 53cb7e5091..a60f1d1e54 100644 --- a/tests/avocado/replay_kernel.py +++ b/tests/avocado/replay_kernel.py @@ -15,7 +15,7 @@ import time from avocado import skip -from avocado import skipIf +from avocado import skipUnless from avocado import skipUnless from avocado_qemu import wait_for_console_pattern from avocado.utils import archive @@ -179,7 +179,8 @@ def test_arm_virt(self): self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1) - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_arm_cubieboard_initrd(self): """ :avocado: tags=arch:arm diff --git a/tests/avocado/reverse_debugging.py b/tests/avocado/reverse_debugging.py index ed04e92bb4..9a468321e5 100644 --- a/tests/avocado/reverse_debugging.py +++ b/tests/avocado/reverse_debugging.py @@ -10,7 +10,7 @@ import os import logging -from avocado import skipIf +from avocado import skipUnless from avocado_qemu import BUILD_DIR from avocado.utils import datadrainer from avocado.utils import gdb @@ -206,7 +206,8 @@ def get_pc(self, g): + self.get_reg_le(g, self.REG_CS) * 0x10 # unidentified gitlab timeout problem - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_x86_64_pc(self): """ :avocado: tags=arch:x86_64 @@ -223,7 +224,8 @@ class ReverseDebugging_AArch64(ReverseDebugging): REG_PC = 32 # unidentified gitlab timeout problem - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_aarch64_virt(self): """ :avocado: tags=arch:aarch64 @@ -247,7 +249,8 @@ class ReverseDebugging_ppc64(ReverseDebugging): REG_PC = 0x40 # unidentified gitlab timeout problem - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_ppc64_pseries(self): """ :avocado: tags=arch:ppc64 @@ -260,7 +263,8 @@ def test_ppc64_pseries(self): self.reverse_debugging() # See https://gitlab.com/qemu-project/qemu/-/issues/1992 - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + def test_ppc64_powernv(self): """ :avocado: tags=arch:ppc64 diff --git a/tests/avocado/smmu.py b/tests/avocado/smmu.py index b3c4de6bf4..05b34418a5 100644 --- a/tests/avocado/smmu.py +++ b/tests/avocado/smmu.py @@ -9,10 +9,11 @@ # later. See the COPYING file in the top-level directory. import os -from avocado import skipIf +from avocado import skipUnless from avocado_qemu import LinuxTest, BUILD_DIR -@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') +@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') + class SMMU(LinuxTest): """ :avocado: tags=accel:kvm diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py index c99bea6c0b..5f859f4e6f 100644 --- a/tests/avocado/tuxrun_baselines.py +++ b/tests/avocado/tuxrun_baselines.py @@ -13,7 +13,7 @@ import time import tempfile -from avocado import skip, skipIf +from avocado import skip, skipUnless from avocado_qemu import QemuSystemTest from avocado_qemu import exec_command, exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern @@ -551,7 +551,7 @@ def test_s390(self): haltmsg="Requesting system halt") # Note: some segfaults caused by unaligned userspace access - @skipIf(os.getenv('GITLAB_CI'), 'Skipping unstable test on GitLab') + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') def test_sh4(self): """ :avocado: tags=arch:sh4 -- 2.39.2 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 2/3] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS 2023-11-30 15:33 ` [PATCH 2/3] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS Alex Bennée @ 2023-11-30 16:06 ` Cédric Le Goater 2023-11-30 16:25 ` Philippe Mathieu-Daudé 2023-11-30 16:40 ` Peter Maydell 2 siblings, 0 replies; 9+ messages in thread From: Cédric Le Goater @ 2023-11-30 16:06 UTC (permalink / raw) To: Alex Bennée, qemu-devel Cc: Eric Farman, Cleber Rosa, Andrew Jeffery, Wainer dos Santos Moschetta, Philippe Mathieu-Daudé, Yoshinori Sato, Eric Auger, qemu-arm, Peter Maydell, Halil Pasic, Pavel Dovgalyuk, Aurelien Jarno, qemu-s390x, Beraldo Leal, Joel Stanley, Paolo Bonzini, Christian Borntraeger, Thomas Huth, Nicholas Piggin On 11/30/23 16:33, Alex Bennée wrote: > It doesn't make sense to have two classes of flaky tests. While it may > take the constrained environment of CI to trigger failures easily it > doesn't mean they don't occasionally happen on developer machines. As > CI is the gating factor to passing there is no point developers > running the tests locally anyway unless they are trying to fix things. > > While we are at it update the language in the docs to discourage the > QEMU_TEST_FLAKY_TESTS becoming a permanent solution. > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > --- > docs/devel/testing.rst | 31 +++++++++++++++--------- > tests/avocado/boot_linux.py | 8 +++--- > tests/avocado/boot_linux_console.py | 5 ++-- > tests/avocado/intel_iommu.py | 5 ++-- > tests/avocado/linux_initrd.py | 5 ++-- > tests/avocado/machine_aspeed.py | 8 +++--- > tests/avocado/machine_mips_malta.py | 8 +++--- > tests/avocado/machine_rx_gdbsim.py | 8 +++--- > tests/avocado/machine_s390_ccw_virtio.py | 2 +- > tests/avocado/replay_kernel.py | 5 ++-- > tests/avocado/reverse_debugging.py | 14 +++++++---- > tests/avocado/smmu.py | 5 ++-- > tests/avocado/tuxrun_baselines.py | 4 +-- > 13 files changed, 67 insertions(+), 41 deletions(-) > > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst > index 22218dbedb..579d1837e0 100644 > --- a/docs/devel/testing.rst > +++ b/docs/devel/testing.rst > @@ -1371,23 +1371,32 @@ conditions. For example, tests that take longer to execute when QEMU is > compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable > has been used to determine whether those tests should run or not. > > -GITLAB_CI > -^^^^^^^^^ > -A number of tests are flagged to not run on the GitLab CI. Usually because > -they proved to the flaky or there are constraints on the CI environment which > -would make them fail. If you encounter a similar situation then use that > -variable as shown on the code snippet below to skip the test: > +QEMU_TEST_FLAKY_TESTS > +^^^^^^^^^^^^^^^^^^^^^ > +Some tests are not working reliably and thus are disabled by default. > +This includes tests that don't run reliably on GitLab's CI which > +usually expose real issues that are rarely seen on developer machines > +due to the constraints of the CI environment. If you encounter a > +similar situation then mark the test as shown on the code snippet > +below: > > .. code:: > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > def test(self): > do_something() > > -QEMU_TEST_FLAKY_TESTS > -^^^^^^^^^^^^^^^^^^^^^ > -Some tests are not working reliably and thus are disabled by default. > -Set this environment variable to enable them. > +Tests should not live in this state forever and should either be fixed > +or eventually removed. If you move a test into this category please > +consider raising a bug to track progress towards a fix. > + > +To run such tests locally you will need to set the environment > +variable. For example: > + > +.. code:: > + > + env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado run \ > + tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg > > Uninstalling Avocado > ~~~~~~~~~~~~~~~~~~~~ > diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py > index be30dcbd58..9e9773e6e1 100644 > --- a/tests/avocado/boot_linux.py > +++ b/tests/avocado/boot_linux.py > @@ -12,7 +12,7 @@ > > from avocado_qemu import LinuxTest, BUILD_DIR > > -from avocado import skipIf > +from avocado import skipUnless > > > class BootLinuxX8664(LinuxTest): > @@ -93,7 +93,8 @@ class BootLinuxPPC64(LinuxTest): > > timeout = 360 > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_pseries_tcg(self): is it because the test is too slow ? If this is the case then we should probably remove. I have never see it fails though. > """ > :avocado: tags=machine:pseries > @@ -111,7 +112,8 @@ class BootLinuxS390X(LinuxTest): > > timeout = 240 > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_s390_ccw_virtio_tcg(self): > """ > :avocado: tags=machine:s390-ccw-virtio > diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py > index 6eab515718..231b4f68e5 100644 > --- a/tests/avocado/boot_linux_console.py > +++ b/tests/avocado/boot_linux_console.py > @@ -15,7 +15,7 @@ > > from avocado import skip > from avocado import skipUnless > -from avocado import skipIf > +from avocado import skipUnless > from avocado_qemu import QemuSystemTest > from avocado_qemu import exec_command > from avocado_qemu import exec_command_and_wait_for_pattern > @@ -1419,7 +1419,8 @@ def test_ppc_mac99(self): > # This test has a 6-10% failure rate on various hosts that look > # like issues with a buggy kernel. As a result we don't want it > # gating releases on Gitlab. > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_sh4_r2d(self): > """ > :avocado: tags=arch:sh4 > diff --git a/tests/avocado/intel_iommu.py b/tests/avocado/intel_iommu.py > index 77635ab56c..2dd11a6346 100644 > --- a/tests/avocado/intel_iommu.py > +++ b/tests/avocado/intel_iommu.py > @@ -9,10 +9,11 @@ > # later. See the COPYING file in the top-level directory. > import os > > -from avocado import skipIf > +from avocado import skipUnless > from avocado_qemu import LinuxTest > > -@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > +@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > class IntelIOMMU(LinuxTest): > """ > :avocado: tags=arch:x86_64 > diff --git a/tests/avocado/linux_initrd.py b/tests/avocado/linux_initrd.py > index ba02e5a563..c40a987bd1 100644 > --- a/tests/avocado/linux_initrd.py > +++ b/tests/avocado/linux_initrd.py > @@ -13,7 +13,7 @@ > import tempfile > > from avocado_qemu import QemuSystemTest > -from avocado import skipIf > +from avocado import skipUnless > > > class LinuxInitrd(QemuSystemTest): > @@ -53,7 +53,8 @@ def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self): > max_size + 1) > self.assertRegex(self.vm.get_log(), expected_msg) > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_with_2gib_file_should_work_with_linux_v4_16(self): > """ > QEMU has supported up to 4 GiB initrd for recent kernel > diff --git a/tests/avocado/machine_aspeed.py b/tests/avocado/machine_aspeed.py > index df31b2a8a8..258fb50c47 100644 > --- a/tests/avocado/machine_aspeed.py > +++ b/tests/avocado/machine_aspeed.py > @@ -18,7 +18,7 @@ > from avocado_qemu import interrupt_interactive_console_until_pattern > from avocado_qemu import has_cmd > from avocado.utils import archive > -from avocado import skipIf > +from avocado import skipUnless > from avocado import skipUnless > > > @@ -311,7 +311,8 @@ def do_test_arm_aspeed_sdk_start(self, image): > self, 'boot', '## Loading kernel from FIT Image') > self.wait_for_console_pattern('Starting kernel ...') > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_arm_ast2500_evb_sdk(self): > """ > :avocado: tags=arch:arm > @@ -329,7 +330,8 @@ def test_arm_ast2500_evb_sdk(self): > self.workdir + '/ast2500-default/image-bmc') > self.wait_for_console_pattern('nodistro.0 ast2500-default ttyS4') > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_arm_ast2600_evb_sdk(self): The Aspeed SDK images are rather heavy and booting them can be slow. Is that the problem ? because I have never seen these tests fail (on fast machines). We should remove in that case. CI can not test all FW images and it should be fast. Thanks, C. > """ > :avocado: tags=arch:arm > diff --git a/tests/avocado/machine_mips_malta.py b/tests/avocado/machine_mips_malta.py > index 9bd54518bf..959dcf5602 100644 > --- a/tests/avocado/machine_mips_malta.py > +++ b/tests/avocado/machine_mips_malta.py > @@ -11,7 +11,7 @@ > import gzip > import logging > > -from avocado import skipIf > +from avocado import skipUnless > from avocado import skipUnless > from avocado.utils import archive > from avocado_qemu import QemuSystemTest > @@ -101,7 +101,8 @@ def test_mips_malta_i6400_framebuffer_logo_1core(self): > """ > self.do_test_i6400_framebuffer_logo(1) > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_mips_malta_i6400_framebuffer_logo_7cores(self): > """ > :avocado: tags=arch:mips64el > @@ -111,7 +112,8 @@ def test_mips_malta_i6400_framebuffer_logo_7cores(self): > """ > self.do_test_i6400_framebuffer_logo(7) > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_mips_malta_i6400_framebuffer_logo_8cores(self): > """ > :avocado: tags=arch:mips64el > diff --git a/tests/avocado/machine_rx_gdbsim.py b/tests/avocado/machine_rx_gdbsim.py > index 6cd8704b01..350a73fbbb 100644 > --- a/tests/avocado/machine_rx_gdbsim.py > +++ b/tests/avocado/machine_rx_gdbsim.py > @@ -10,7 +10,7 @@ > > import os > > -from avocado import skipIf > +from avocado import skipUnless > from avocado_qemu import QemuSystemTest > from avocado_qemu import exec_command_and_wait_for_pattern > from avocado_qemu import wait_for_console_pattern > @@ -22,7 +22,8 @@ class RxGdbSimMachine(QemuSystemTest): > timeout = 30 > KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 ' > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_uboot(self): > """ > U-Boot and checks that the console is operational. > @@ -46,7 +47,8 @@ def test_uboot(self): > # FIXME limit baudrate on chardev, else we type too fast > #exec_command_and_wait_for_pattern(self, 'version', gcc_version) > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_linux_sash(self): > """ > Boots a Linux kernel and checks that the console is operational. > diff --git a/tests/avocado/machine_s390_ccw_virtio.py b/tests/avocado/machine_s390_ccw_virtio.py > index ffd914ded9..61e75d8f9e 100644 > --- a/tests/avocado/machine_s390_ccw_virtio.py > +++ b/tests/avocado/machine_s390_ccw_virtio.py > @@ -12,7 +12,7 @@ > import os > import tempfile > > -from avocado import skipIf > +from avocado import skipUnless > from avocado_qemu import QemuSystemTest > from avocado_qemu import exec_command_and_wait_for_pattern > from avocado_qemu import wait_for_console_pattern > diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py > index 53cb7e5091..a60f1d1e54 100644 > --- a/tests/avocado/replay_kernel.py > +++ b/tests/avocado/replay_kernel.py > @@ -15,7 +15,7 @@ > import time > > from avocado import skip > -from avocado import skipIf > +from avocado import skipUnless > from avocado import skipUnless > from avocado_qemu import wait_for_console_pattern > from avocado.utils import archive > @@ -179,7 +179,8 @@ def test_arm_virt(self): > > self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=1) > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_arm_cubieboard_initrd(self): > """ > :avocado: tags=arch:arm > diff --git a/tests/avocado/reverse_debugging.py b/tests/avocado/reverse_debugging.py > index ed04e92bb4..9a468321e5 100644 > --- a/tests/avocado/reverse_debugging.py > +++ b/tests/avocado/reverse_debugging.py > @@ -10,7 +10,7 @@ > import os > import logging > > -from avocado import skipIf > +from avocado import skipUnless > from avocado_qemu import BUILD_DIR > from avocado.utils import datadrainer > from avocado.utils import gdb > @@ -206,7 +206,8 @@ def get_pc(self, g): > + self.get_reg_le(g, self.REG_CS) * 0x10 > > # unidentified gitlab timeout problem > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_x86_64_pc(self): > """ > :avocado: tags=arch:x86_64 > @@ -223,7 +224,8 @@ class ReverseDebugging_AArch64(ReverseDebugging): > REG_PC = 32 > > # unidentified gitlab timeout problem > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_aarch64_virt(self): > """ > :avocado: tags=arch:aarch64 > @@ -247,7 +249,8 @@ class ReverseDebugging_ppc64(ReverseDebugging): > REG_PC = 0x40 > > # unidentified gitlab timeout problem > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_ppc64_pseries(self): > """ > :avocado: tags=arch:ppc64 > @@ -260,7 +263,8 @@ def test_ppc64_pseries(self): > self.reverse_debugging() > > # See https://gitlab.com/qemu-project/qemu/-/issues/1992 > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > def test_ppc64_powernv(self): > """ > :avocado: tags=arch:ppc64 > diff --git a/tests/avocado/smmu.py b/tests/avocado/smmu.py > index b3c4de6bf4..05b34418a5 100644 > --- a/tests/avocado/smmu.py > +++ b/tests/avocado/smmu.py > @@ -9,10 +9,11 @@ > # later. See the COPYING file in the top-level directory. > import os > > -from avocado import skipIf > +from avocado import skipUnless > from avocado_qemu import LinuxTest, BUILD_DIR > > -@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > +@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > + > class SMMU(LinuxTest): > """ > :avocado: tags=accel:kvm > diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py > index c99bea6c0b..5f859f4e6f 100644 > --- a/tests/avocado/tuxrun_baselines.py > +++ b/tests/avocado/tuxrun_baselines.py > @@ -13,7 +13,7 @@ > import time > import tempfile > > -from avocado import skip, skipIf > +from avocado import skip, skipUnless > from avocado_qemu import QemuSystemTest > from avocado_qemu import exec_command, exec_command_and_wait_for_pattern > from avocado_qemu import wait_for_console_pattern > @@ -551,7 +551,7 @@ def test_s390(self): > haltmsg="Requesting system halt") > > # Note: some segfaults caused by unaligned userspace access > - @skipIf(os.getenv('GITLAB_CI'), 'Skipping unstable test on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > def test_sh4(self): > """ > :avocado: tags=arch:sh4 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/3] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS 2023-11-30 15:33 ` [PATCH 2/3] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS Alex Bennée 2023-11-30 16:06 ` Cédric Le Goater @ 2023-11-30 16:25 ` Philippe Mathieu-Daudé 2023-11-30 16:40 ` Peter Maydell 2 siblings, 0 replies; 9+ messages in thread From: Philippe Mathieu-Daudé @ 2023-11-30 16:25 UTC (permalink / raw) To: Alex Bennée, qemu-devel Cc: Eric Farman, Cleber Rosa, Andrew Jeffery, Wainer dos Santos Moschetta, Yoshinori Sato, Cédric Le Goater, Eric Auger, qemu-arm, Peter Maydell, Halil Pasic, Pavel Dovgalyuk, Aurelien Jarno, qemu-s390x, Beraldo Leal, Joel Stanley, Paolo Bonzini, Christian Borntraeger, Thomas Huth On 30/11/23 16:33, Alex Bennée wrote: > It doesn't make sense to have two classes of flaky tests. While it may > take the constrained environment of CI to trigger failures easily it > doesn't mean they don't occasionally happen on developer machines. As > CI is the gating factor to passing there is no point developers > running the tests locally anyway unless they are trying to fix things. > > While we are at it update the language in the docs to discourage the > QEMU_TEST_FLAKY_TESTS becoming a permanent solution. > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > --- > docs/devel/testing.rst | 31 +++++++++++++++--------- > tests/avocado/boot_linux.py | 8 +++--- > tests/avocado/boot_linux_console.py | 5 ++-- > tests/avocado/intel_iommu.py | 5 ++-- > tests/avocado/linux_initrd.py | 5 ++-- > tests/avocado/machine_aspeed.py | 8 +++--- > tests/avocado/machine_mips_malta.py | 8 +++--- > tests/avocado/machine_rx_gdbsim.py | 8 +++--- > tests/avocado/machine_s390_ccw_virtio.py | 2 +- > tests/avocado/replay_kernel.py | 5 ++-- > tests/avocado/reverse_debugging.py | 14 +++++++---- > tests/avocado/smmu.py | 5 ++-- > tests/avocado/tuxrun_baselines.py | 4 +-- > 13 files changed, 67 insertions(+), 41 deletions(-) > diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py > index be30dcbd58..9e9773e6e1 100644 > --- a/tests/avocado/boot_linux.py > +++ b/tests/avocado/boot_linux.py > @@ -12,7 +12,7 @@ > > from avocado_qemu import LinuxTest, BUILD_DIR > > -from avocado import skipIf > +from avocado import skipUnless > > > class BootLinuxX8664(LinuxTest): > @@ -93,7 +93,8 @@ class BootLinuxPPC64(LinuxTest): > > timeout = 360 > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') Later we could move that to a @flakyTest decorator around QemuBaseTest. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> > + > def test_pseries_tcg(self): > """ > :avocado: tags=machine:pseries ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/3] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS 2023-11-30 15:33 ` [PATCH 2/3] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS Alex Bennée 2023-11-30 16:06 ` Cédric Le Goater 2023-11-30 16:25 ` Philippe Mathieu-Daudé @ 2023-11-30 16:40 ` Peter Maydell 2023-11-30 16:59 ` Alex Bennée 2 siblings, 1 reply; 9+ messages in thread From: Peter Maydell @ 2023-11-30 16:40 UTC (permalink / raw) To: Alex Bennée Cc: qemu-devel, Eric Farman, Cleber Rosa, Andrew Jeffery, Wainer dos Santos Moschetta, Philippe Mathieu-Daudé, Yoshinori Sato, Cédric Le Goater, Eric Auger, qemu-arm, Halil Pasic, Pavel Dovgalyuk, Aurelien Jarno, qemu-s390x, Beraldo Leal, Joel Stanley, Paolo Bonzini, Christian Borntraeger, Thomas Huth On Thu, 30 Nov 2023 at 15:33, Alex Bennée <alex.bennee@linaro.org> wrote: > > It doesn't make sense to have two classes of flaky tests. While it may > take the constrained environment of CI to trigger failures easily it > doesn't mean they don't occasionally happen on developer machines. As > CI is the gating factor to passing there is no point developers > running the tests locally anyway unless they are trying to fix things. > > While we are at it update the language in the docs to discourage the > QEMU_TEST_FLAKY_TESTS becoming a permanent solution. > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > --- > docs/devel/testing.rst | 31 +++++++++++++++--------- > tests/avocado/boot_linux.py | 8 +++--- > tests/avocado/boot_linux_console.py | 5 ++-- > tests/avocado/intel_iommu.py | 5 ++-- > tests/avocado/linux_initrd.py | 5 ++-- > tests/avocado/machine_aspeed.py | 8 +++--- > tests/avocado/machine_mips_malta.py | 8 +++--- > tests/avocado/machine_rx_gdbsim.py | 8 +++--- > tests/avocado/machine_s390_ccw_virtio.py | 2 +- > tests/avocado/replay_kernel.py | 5 ++-- > tests/avocado/reverse_debugging.py | 14 +++++++---- > tests/avocado/smmu.py | 5 ++-- > tests/avocado/tuxrun_baselines.py | 4 +-- > 13 files changed, 67 insertions(+), 41 deletions(-) > > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst > index 22218dbedb..579d1837e0 100644 > --- a/docs/devel/testing.rst > +++ b/docs/devel/testing.rst > @@ -1371,23 +1371,32 @@ conditions. For example, tests that take longer to execute when QEMU is > compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable > has been used to determine whether those tests should run or not. > > -GITLAB_CI > -^^^^^^^^^ > -A number of tests are flagged to not run on the GitLab CI. Usually because > -they proved to the flaky or there are constraints on the CI environment which > -would make them fail. If you encounter a similar situation then use that > -variable as shown on the code snippet below to skip the test: > +QEMU_TEST_FLAKY_TESTS > +^^^^^^^^^^^^^^^^^^^^^ > +Some tests are not working reliably and thus are disabled by default. > +This includes tests that don't run reliably on GitLab's CI which > +usually expose real issues that are rarely seen on developer machines > +due to the constraints of the CI environment. If you encounter a > +similar situation then mark the test as shown on the code snippet > +below: > > .. code:: > > - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') > + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') > def test(self): > do_something() Can we also say here that when marking a test as flaky: * we should raise a github issue giving details of what goes wrong * the URL of that issue should be in a comment above the @skipUnless line ? That way we have a history of why we disabled the test and we might even manage to fix it some day, in which case we'll know we are able to unmark it as flaky... > +To run such tests locally you will need to set the environment > +variable. For example: > + > +.. code:: > + > + env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado run \ > + tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg The "env" here is unnecessary (assuming a standard Posix shell); "VAR=value foo" will run "foo" with VAR set to 'value' only for the duration of that command. thanks -- PMM ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/3] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS 2023-11-30 16:40 ` Peter Maydell @ 2023-11-30 16:59 ` Alex Bennée 0 siblings, 0 replies; 9+ messages in thread From: Alex Bennée @ 2023-11-30 16:59 UTC (permalink / raw) To: Peter Maydell Cc: qemu-devel, Eric Farman, Cleber Rosa, Andrew Jeffery, Wainer dos Santos Moschetta, Philippe Mathieu-Daudé, Yoshinori Sato, Cédric Le Goater, Eric Auger, qemu-arm, Halil Pasic, Pavel Dovgalyuk, Aurelien Jarno, qemu-s390x, Beraldo Leal, Joel Stanley, Paolo Bonzini, Christian Borntraeger, Thomas Huth Peter Maydell <peter.maydell@linaro.org> writes: > On Thu, 30 Nov 2023 at 15:33, Alex Bennée <alex.bennee@linaro.org> wrote: >> >> It doesn't make sense to have two classes of flaky tests. While it may >> take the constrained environment of CI to trigger failures easily it >> doesn't mean they don't occasionally happen on developer machines. As >> CI is the gating factor to passing there is no point developers >> running the tests locally anyway unless they are trying to fix things. >> >> While we are at it update the language in the docs to discourage the >> QEMU_TEST_FLAKY_TESTS becoming a permanent solution. >> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> >> --- >> docs/devel/testing.rst | 31 +++++++++++++++--------- >> tests/avocado/boot_linux.py | 8 +++--- >> tests/avocado/boot_linux_console.py | 5 ++-- >> tests/avocado/intel_iommu.py | 5 ++-- >> tests/avocado/linux_initrd.py | 5 ++-- >> tests/avocado/machine_aspeed.py | 8 +++--- >> tests/avocado/machine_mips_malta.py | 8 +++--- >> tests/avocado/machine_rx_gdbsim.py | 8 +++--- >> tests/avocado/machine_s390_ccw_virtio.py | 2 +- >> tests/avocado/replay_kernel.py | 5 ++-- >> tests/avocado/reverse_debugging.py | 14 +++++++---- >> tests/avocado/smmu.py | 5 ++-- >> tests/avocado/tuxrun_baselines.py | 4 +-- >> 13 files changed, 67 insertions(+), 41 deletions(-) >> >> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst >> index 22218dbedb..579d1837e0 100644 >> --- a/docs/devel/testing.rst >> +++ b/docs/devel/testing.rst >> @@ -1371,23 +1371,32 @@ conditions. For example, tests that take longer to execute when QEMU is >> compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable >> has been used to determine whether those tests should run or not. >> >> -GITLAB_CI >> -^^^^^^^^^ >> -A number of tests are flagged to not run on the GitLab CI. Usually because >> -they proved to the flaky or there are constraints on the CI environment which >> -would make them fail. If you encounter a similar situation then use that >> -variable as shown on the code snippet below to skip the test: >> +QEMU_TEST_FLAKY_TESTS >> +^^^^^^^^^^^^^^^^^^^^^ >> +Some tests are not working reliably and thus are disabled by default. >> +This includes tests that don't run reliably on GitLab's CI which >> +usually expose real issues that are rarely seen on developer machines >> +due to the constraints of the CI environment. If you encounter a >> +similar situation then mark the test as shown on the code snippet >> +below: >> >> .. code:: >> >> - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') >> + @skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab') >> def test(self): >> do_something() > > Can we also say here that when marking a test as flaky: > * we should raise a github issue giving details of what goes wrong > * the URL of that issue should be in a comment above the @skipUnless > line ? > > That way we have a history of why we disabled the test and we > might even manage to fix it some day, in which case we'll know > we are able to unmark it as flaky... Good idea, I'll update. > >> +To run such tests locally you will need to set the environment >> +variable. For example: >> + >> +.. code:: >> + >> + env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado run \ >> + tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg > > The "env" here is unnecessary (assuming a standard Posix shell); > "VAR=value foo" will run "foo" with VAR set to 'value' only > for the duration of that command. Ahh I habit I've picked up from running a fish shell. Do we want to assume everyone's interactive shell is Posix compatible? > > thanks > -- PMM -- Alex Bennée Virtualisation Tech Lead @ Linaro ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 3/3] gitlab: add optional job to run flaky avocado tests 2023-11-30 15:33 [PATCH 0/3] final fixes for 8.2 Alex Bennée 2023-11-30 15:33 ` [PATCH 1/3] gdbstub: use a better signal when we halt for IO reasons Alex Bennée 2023-11-30 15:33 ` [PATCH 2/3] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS Alex Bennée @ 2023-11-30 15:33 ` Alex Bennée 2023-11-30 16:05 ` [PATCH 0/3] final fixes for 8.2 Richard Henderson 3 siblings, 0 replies; 9+ messages in thread From: Alex Bennée @ 2023-11-30 15:33 UTC (permalink / raw) To: qemu-devel Cc: Eric Farman, Cleber Rosa, Andrew Jeffery, Wainer dos Santos Moschetta, Philippe Mathieu-Daudé, Yoshinori Sato, Cédric Le Goater, Alex Bennée, Eric Auger, qemu-arm, Peter Maydell, Halil Pasic, Pavel Dovgalyuk, Aurelien Jarno, qemu-s390x, Beraldo Leal, Joel Stanley, Paolo Bonzini, Christian Borntraeger, Thomas Huth One problem with flaky tests is they often only fail under CI conditions which makes it hard to debug. We add an optional allow_fail job so developers can trigger the only the flaky tests in the CI environment if they are debugging. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> --- docs/devel/testing.rst | 15 ++++++------ .gitlab-ci.d/buildtest.yml | 30 ++++++++++++++++++++++++ tests/avocado/boot_linux.py | 2 ++ tests/avocado/boot_linux_console.py | 1 + tests/avocado/intel_iommu.py | 1 + tests/avocado/linux_initrd.py | 2 ++ tests/avocado/machine_aspeed.py | 2 ++ tests/avocado/machine_mips_malta.py | 2 ++ tests/avocado/machine_rx_gdbsim.py | 2 ++ tests/avocado/machine_s390_ccw_virtio.py | 1 + tests/avocado/replay_kernel.py | 2 ++ tests/avocado/reverse_debugging.py | 2 ++ tests/avocado/smmu.py | 1 + tests/avocado/tuxrun_baselines.py | 1 + 14 files changed, 57 insertions(+), 7 deletions(-) diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 579d1837e0..fec8f06c09 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -1386,17 +1386,18 @@ below: def test(self): do_something() +You can also add ``:avocado: tags=flaky`` to the test meta-data so +only the flaky tests can be run as a group: + +.. code:: + + env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado \ + run tests/avocado -filter-by-tags=flaky + Tests should not live in this state forever and should either be fixed or eventually removed. If you move a test into this category please consider raising a bug to track progress towards a fix. -To run such tests locally you will need to set the environment -variable. For example: - -.. code:: - - env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado run \ - tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg Uninstalling Avocado ~~~~~~~~~~~~~~~~~~~~ diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml index 7f9af83b10..1624518d6c 100644 --- a/.gitlab-ci.d/buildtest.yml +++ b/.gitlab-ci.d/buildtest.yml @@ -217,6 +217,36 @@ avocado-system-opensuse: MAKE_CHECK_ARGS: check-avocado AVOCADO_TAGS: arch:s390x arch:x86_64 arch:aarch64 +# +# Flaky tests. We don't run these by default and they are allow fail +# but often the CI system is the only way to trigger the failures. +# + +build-system-flaky: + extends: + - .native_build_job_template + - .native_build_artifact_template + needs: + job: amd64-debian-container + variables: + IMAGE: debian + QEMU_JOB_OPTIONAL: 1 + TARGETS: arm-softmmu mips64el-softmmu rx-softmmu s390x-softmmu + sh4-softmmu x86_64-softmmu + MAKE_CHECK_ARGS: check-build + +avocado-system-flaky: + extends: .avocado_test_job_template + needs: + - job: build-system-flaky + artifacts: true + allow_failure: true + variables: + IMAGE: debian + MAKE_CHECK_ARGS: check-avocado + QEMU_JOB_OPTIONAL: 1 + QEMU_TEST_FLAKY_TESTS: 1 + AVOCADO_TAGS: flaky # This jobs explicitly disable TCG (--disable-tcg), KVM is detected by # the configure script. The container doesn't contain Xen headers so diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py index 9e9773e6e1..7c4769904e 100644 --- a/tests/avocado/boot_linux.py +++ b/tests/avocado/boot_linux.py @@ -99,6 +99,7 @@ def test_pseries_tcg(self): """ :avocado: tags=machine:pseries :avocado: tags=accel:tcg + :avocado: tags=flaky """ self.require_accelerator("tcg") self.vm.add_args("-accel", "tcg") @@ -118,6 +119,7 @@ def test_s390_ccw_virtio_tcg(self): """ :avocado: tags=machine:s390-ccw-virtio :avocado: tags=accel:tcg + :avocado: tags=flaky """ self.require_accelerator("tcg") self.vm.add_args("-accel", "tcg") diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py index 231b4f68e5..3f0180e1f8 100644 --- a/tests/avocado/boot_linux_console.py +++ b/tests/avocado/boot_linux_console.py @@ -1425,6 +1425,7 @@ def test_sh4_r2d(self): """ :avocado: tags=arch:sh4 :avocado: tags=machine:r2d + :avocado: tags=flaky """ tar_hash = 'fe06a4fd8ccbf2e27928d64472939d47829d4c7e' self.vm.add_args('-append', 'console=ttySC1') diff --git a/tests/avocado/intel_iommu.py b/tests/avocado/intel_iommu.py index 2dd11a6346..f04ee1cf9d 100644 --- a/tests/avocado/intel_iommu.py +++ b/tests/avocado/intel_iommu.py @@ -22,6 +22,7 @@ class IntelIOMMU(LinuxTest): :avocado: tags=machine:q35 :avocado: tags=accel:kvm :avocado: tags=intel_iommu + :avocado: tags=flaky """ IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on' diff --git a/tests/avocado/linux_initrd.py b/tests/avocado/linux_initrd.py index c40a987bd1..aad5b19bd9 100644 --- a/tests/avocado/linux_initrd.py +++ b/tests/avocado/linux_initrd.py @@ -57,6 +57,8 @@ def test_with_2gib_file_should_exit_error_msg_with_linux_v3_6(self): def test_with_2gib_file_should_work_with_linux_v4_16(self): """ + :avocado: tags=flaky + QEMU has supported up to 4 GiB initrd for recent kernel Expect guest can reach 'Unpacking initramfs...' """ diff --git a/tests/avocado/machine_aspeed.py b/tests/avocado/machine_aspeed.py index 258fb50c47..6fa5459a07 100644 --- a/tests/avocado/machine_aspeed.py +++ b/tests/avocado/machine_aspeed.py @@ -317,6 +317,7 @@ def test_arm_ast2500_evb_sdk(self): """ :avocado: tags=arch:arm :avocado: tags=machine:ast2500-evb + :avocado: tags=flaky """ image_url = ('https://github.com/AspeedTech-BMC/openbmc/releases/' @@ -336,6 +337,7 @@ def test_arm_ast2600_evb_sdk(self): """ :avocado: tags=arch:arm :avocado: tags=machine:ast2600-evb + :avocado: tags=flaky """ image_url = ('https://github.com/AspeedTech-BMC/openbmc/releases/' diff --git a/tests/avocado/machine_mips_malta.py b/tests/avocado/machine_mips_malta.py index 959dcf5602..99bee49e9a 100644 --- a/tests/avocado/machine_mips_malta.py +++ b/tests/avocado/machine_mips_malta.py @@ -109,6 +109,7 @@ def test_mips_malta_i6400_framebuffer_logo_7cores(self): :avocado: tags=machine:malta :avocado: tags=cpu:I6400 :avocado: tags=mips:smp + :avocado: tags=flaky """ self.do_test_i6400_framebuffer_logo(7) @@ -120,6 +121,7 @@ def test_mips_malta_i6400_framebuffer_logo_8cores(self): :avocado: tags=machine:malta :avocado: tags=cpu:I6400 :avocado: tags=mips:smp + :avocado: tags=flaky """ self.do_test_i6400_framebuffer_logo(8) diff --git a/tests/avocado/machine_rx_gdbsim.py b/tests/avocado/machine_rx_gdbsim.py index 350a73fbbb..412a7a5089 100644 --- a/tests/avocado/machine_rx_gdbsim.py +++ b/tests/avocado/machine_rx_gdbsim.py @@ -31,6 +31,7 @@ def test_uboot(self): :avocado: tags=arch:rx :avocado: tags=machine:gdbsim-r5f562n8 :avocado: tags=endian:little + :avocado: tags=flaky """ uboot_url = ('https://acc.dl.osdn.jp/users/23/23888/u-boot.bin.gz') uboot_hash = '9b78dbd43b40b2526848c0b1ce9de02c24f4dcdb' @@ -56,6 +57,7 @@ def test_linux_sash(self): :avocado: tags=arch:rx :avocado: tags=machine:gdbsim-r5f562n7 :avocado: tags=endian:little + :avocado: tags=flaky """ dtb_url = ('https://acc.dl.osdn.jp/users/23/23887/rx-virt.dtb') dtb_hash = '7b4e4e2c71905da44e86ce47adee2210b026ac18' diff --git a/tests/avocado/machine_s390_ccw_virtio.py b/tests/avocado/machine_s390_ccw_virtio.py index 61e75d8f9e..26e938c9e9 100644 --- a/tests/avocado/machine_s390_ccw_virtio.py +++ b/tests/avocado/machine_s390_ccw_virtio.py @@ -167,6 +167,7 @@ def test_s390x_fedora(self): :avocado: tags=device:virtio-gpu :avocado: tags=device:virtio-crypto :avocado: tags=device:virtio-net + :avocado: tags=flaky """ kernel_url = ('https://archives.fedoraproject.org/pub/archive' diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py index a60f1d1e54..010d7644c2 100644 --- a/tests/avocado/replay_kernel.py +++ b/tests/avocado/replay_kernel.py @@ -87,6 +87,7 @@ def test_x86_64_pc(self): """ :avocado: tags=arch:x86_64 :avocado: tags=machine:pc + :avocado: tags=flaky """ kernel_url = ('https://archives.fedoraproject.org/pub/archive/fedora' '/linux/releases/29/Everything/x86_64/os/images/pxeboot' @@ -185,6 +186,7 @@ def test_arm_cubieboard_initrd(self): """ :avocado: tags=arch:arm :avocado: tags=machine:cubieboard + :avocado: tags=flaky """ deb_url = ('https://apt.armbian.com/pool/main/l/' 'linux-5.10.16-sunxi/linux-image-current-sunxi_21.02.2_armhf.deb') diff --git a/tests/avocado/reverse_debugging.py b/tests/avocado/reverse_debugging.py index 9a468321e5..4cce5a5598 100644 --- a/tests/avocado/reverse_debugging.py +++ b/tests/avocado/reverse_debugging.py @@ -255,6 +255,7 @@ def test_ppc64_pseries(self): """ :avocado: tags=arch:ppc64 :avocado: tags=machine:pseries + :avocado: tags=flaky """ # SLOF branches back to its entry point, which causes this test # to take the 'hit a breakpoint again' path. That's not a problem, @@ -269,6 +270,7 @@ def test_ppc64_powernv(self): """ :avocado: tags=arch:ppc64 :avocado: tags=machine:powernv + :avocado: tags=flaky """ self.endian_is_le = False self.reverse_debugging() diff --git a/tests/avocado/smmu.py b/tests/avocado/smmu.py index 05b34418a5..21ff030ca7 100644 --- a/tests/avocado/smmu.py +++ b/tests/avocado/smmu.py @@ -22,6 +22,7 @@ class SMMU(LinuxTest): :avocado: tags=machine:virt :avocado: tags=distro:fedora :avocado: tags=smmu + :avocado: tags=flaky """ IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on' diff --git a/tests/avocado/tuxrun_baselines.py b/tests/avocado/tuxrun_baselines.py index 5f859f4e6f..a936a3b780 100644 --- a/tests/avocado/tuxrun_baselines.py +++ b/tests/avocado/tuxrun_baselines.py @@ -561,6 +561,7 @@ def test_sh4(self): :avocado: tags=image:zImage :avocado: tags=root:sda :avocado: tags=console:ttySC1 + :avocado: tags=flaky """ sums = { "rootfs.ext4.zst" : "3592a7a3d5a641e8b9821449e77bc43c9904a56c30d45da0694349cfd86743fd", -- 2.39.2 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 0/3] final fixes for 8.2 2023-11-30 15:33 [PATCH 0/3] final fixes for 8.2 Alex Bennée ` (2 preceding siblings ...) 2023-11-30 15:33 ` [PATCH 3/3] gitlab: add optional job to run flaky avocado tests Alex Bennée @ 2023-11-30 16:05 ` Richard Henderson 3 siblings, 0 replies; 9+ messages in thread From: Richard Henderson @ 2023-11-30 16:05 UTC (permalink / raw) To: Alex Bennée, qemu-devel Cc: Eric Farman, Cleber Rosa, Andrew Jeffery, Wainer dos Santos Moschetta, Philippe Mathieu-Daudé, Yoshinori Sato, Cédric Le Goater, Eric Auger, qemu-arm, Peter Maydell, Halil Pasic, Pavel Dovgalyuk, Aurelien Jarno, qemu-s390x, Beraldo Leal, Joel Stanley, Paolo Bonzini, Christian Borntraeger, Thomas Huth On 11/30/23 09:33, Alex Bennée wrote: > Alex Bennée (3): > gdbstub: use a better signal when we halt for IO reasons > docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS > gitlab: add optional job to run flaky avocado tests Reviewed-by: Richard Henderson <richard.henderson@linaro.org> r~ ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2023-11-30 17:18 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-11-30 15:33 [PATCH 0/3] final fixes for 8.2 Alex Bennée 2023-11-30 15:33 ` [PATCH 1/3] gdbstub: use a better signal when we halt for IO reasons Alex Bennée 2023-11-30 15:33 ` [PATCH 2/3] docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS Alex Bennée 2023-11-30 16:06 ` Cédric Le Goater 2023-11-30 16:25 ` Philippe Mathieu-Daudé 2023-11-30 16:40 ` Peter Maydell 2023-11-30 16:59 ` Alex Bennée 2023-11-30 15:33 ` [PATCH 3/3] gitlab: add optional job to run flaky avocado tests Alex Bennée 2023-11-30 16:05 ` [PATCH 0/3] final fixes for 8.2 Richard Henderson
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).