Re: [PATCH] gitlab: remove unreliable avocado CI jobs

[Thomas Huth <thuth@redhat.com>]

On 12/09/2023 17.06, Stefan Hajnoczi wrote:
> The avocado-system-alpine, avocado-system-fedora, and
> avocado-system-ubuntu jobs are unreliable. I identified them while
> looking over CI failures from the past week:
> https://gitlab.com/qemu-project/qemu/-/jobs/5058610614
> https://gitlab.com/qemu-project/qemu/-/jobs/5058610654
> https://gitlab.com/qemu-project/qemu/-/jobs/5030428571
> 
> Thomas Huth suggest on IRC today that there may be a legitimate failure
> in there:
> 
>    th_huth: f4bug, yes, seems like it does not start at all correctly on
>    alpine anymore ... and it's broken since ~ 2 weeks already, so if nobody
>    noticed this by now, this is worrying
> 
> It crept in because the jobs were already unreliable.
> 
> I don't know how to interpret the job output, so all I can do is to
> propose removing these jobs. A useful CI job has two outcomes: pass or
> fail. Timeouts and other in-between states are not useful because they
> require constant triaging by someone who understands the details of the
> tests and they can occur when run against pull requests that have
> nothing to do with the area covered by the test.
> 
> Hopefully test owners will be able to identify the root causes and solve
> them so that these jobs can stay. In their current state the jobs are
> not useful since I cannot cannot tell whether job failures are real or
> just intermittent when merging qemu.git pull requests.
> 
> If you are a test owner, please take a look.
> 
> It is likely that other avocado-system-* CI jobs have similar failures
> from time to time, but I'll leave them as long as they are passing.
> 
> Buglink: https://gitlab.com/qemu-project/qemu/-/issues/1884
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>   .gitlab-ci.d/buildtest.yml | 27 ---------------------------
>   1 file changed, 27 deletions(-)
> 
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index aee9101507..83ce448c4d 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -22,15 +22,6 @@ check-system-alpine:
>       IMAGE: alpine
>       MAKE_CHECK_ARGS: check-unit check-qtest
>   
> -avocado-system-alpine:
> -  extends: .avocado_test_job_template
> -  needs:
> -    - job: build-system-alpine
> -      artifacts: true
> -  variables:
> -    IMAGE: alpine
> -    MAKE_CHECK_ARGS: check-avocado

Please don't remove the whole job! Just disable the failing tests within the job, e.g.:

diff --git a/tests/avocado/replay_kernel.py b/tests/avocado/replay_kernel.py
--- a/tests/avocado/replay_kernel.py
+++ b/tests/avocado/replay_kernel.py
@@ -503,6 +503,7 @@ def do_test_mips_malta32el_nanomips(self, kernel_path_xz):
          console_pattern = 'Kernel command line: %s' % kernel_command_line
          self.run_rr(kernel_path, kernel_command_line, console_pattern, shift=5)
  
+    @skipIf(os.getenv('GITLAB_CI'), 'Skipping unstable test on GitLab')
      def test_mips_malta32el_nanomips_4k(self):
          """
          :avocado: tags=arch:mipsel
@@ -517,6 +518,7 @@ def test_mips_malta32el_nanomips_4k(self):
          kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
          self.do_test_mips_malta32el_nanomips(kernel_path_xz)
  
+    @skipIf(os.getenv('GITLAB_CI'), 'Skipping unstable test on GitLab')
      def test_mips_malta32el_nanomips_16k_up(self):
          """
          :avocado: tags=arch:mipsel
@@ -531,6 +533,7 @@ def test_mips_malta32el_nanomips_16k_up(self):
          kernel_path_xz = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
          self.do_test_mips_malta32el_nanomips(kernel_path_xz)
  
+    @skipIf(os.getenv('GITLAB_CI'), 'Skipping unstable test on GitLab')
      def test_mips_malta32el_nanomips_64k_dbg(self):
          """
          :avocado: tags=arch:mipsel