Re: [RFC PATCH-for-5.2] gitlab-ci: Do not automatically run Avocado integration tests anymore

[Wainer dos Santos Moschetta <wainersm@redhat.com>]

Hi Phil,

On 11/27/20 2:41 PM, Philippe Mathieu-Daudé wrote:
> We lately realized that the Avocado framework was not designed
> to be regularly run on CI environments. Therefore, as of 5.2
> we deprecate the gitlab-ci jobs using Avocado. To not disrupt
> current users, it is possible to keep the current behavior by
> setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable
> (see [*]).


Could you please detail where have you seen the avocado based tests 
failing that much to justify their removal of CI?

See below some data that I generated from the pipelines for 
https://gitlab.com/qemu-project/qemu/. It seems that jobs which failed 
due timeout are marked as 'skip' and that explain why some pipelines 
don't have a list of failed jobs.

---

Failed 21 out of 99
Pipeline -- 221617704 (staging)
check-patch
Pipeline -- 219276598 (master)
pages
Pipeline -- 218121424 (master)
Pipeline -- 217995873 (staging)
acceptance-system-fedora
Pipeline -- 217995872 (master)
build-tci
Pipeline -- 217551771 (v5.2.0-rc2)
Pipeline -- 217503505 (master)
Pipeline -- 217362429 (master)
Pipeline -- 217328707 (master)
check-crypto-old-nettle
Pipeline -- 216770735 (staging)
acceptance-system-fedora
check-patch
Pipeline -- 215772908 (master)
Pipeline -- 215715025 (staging)
check-crypto-old-gcrypt
Pipeline -- 215715024 (master)
check-crypto-old-nettle
Pipeline -- 214944520 (master)
acceptance-system-debian
Pipeline -- 214460230 (v5.2.0-rc1)
Pipeline -- 214438601 (master)
check-crypto-only-gnutls
Pipeline -- 214273938 (master)
Pipeline -- 214183970 (master)
Pipeline -- 214140305 (master)
Pipeline -- 213892224 (master)
Pipeline -- 213871132 (master)
>  From now on, using these jobs (or adding new tests to them)
> is strongly discouraged.
>
> Tests based on Avocado will be ported to new job schemes during
> the next releases, with better documentation and templates.
>
> [*] https://docs.gitlab.com/ee/ci/variables/README.html#create-a-custom-variable-in-the-ui
>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>   .gitlab-ci.yml | 9 +++++++++
>   1 file changed, 9 insertions(+)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index d0173e82b16..2674407cd13 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -66,6 +66,15 @@ include:
>       - cd build
>       - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP", "CANCEL")]' | xargs cat
>       - du -chs ${CI_PROJECT_DIR}/avocado-cache
> +  rules:
> +  # As of QEMU 5.2, Avocado is not yet ready to run in CI environments, therefore
> +  # the jobs based on this template are not run automatically (except if the user
> +  # explicitly sets the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE environment
> +  # variable). Adding new jobs on top of this template is strongly discouraged.
> +  - if: $QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE == null
> +    when: manual
> +    allow_failure: true
> +  - when: always
>   
>   build-system-ubuntu:
>     <<: *native_build_job_definition


The script I used to generate the above metrics:

$ pip install --user python-gitlab

$ cat gitlab-stats
#!/bin/env python3

import gitlab

gl = gitlab.Gitlab('http://gitlab.com')
qemu_project = gl.projects.get(11167699)
pipelines = qemu_project.pipelines.list()

total=0
failed=[]
for pipeline in qemu_project.pipelines.list(page=1, per_page=300):
     if pipeline.status == 'running':
         continue
     elif pipeline.status == 'failed':
         failed.append(pipeline)
     total += 1

print("Failed %d out of %d" % (len(failed), total))
for pipeline in failed:
     print('Pipeline -- %d (%s)' % (pipeline.id, pipeline.ref))
     jobs_failed=[]
     for job in pipeline.jobs.list():
         if job.status == 'failed':
             print(job.name)