Re: [PATCH v3 24/30] gitlab: add acceptance testing to system builds

[Thomas Huth <thuth@redhat.com>]

On 26/06/2020 20.13, Alex Bennée wrote:
> As part of migrating things from Travis to GitLab add the acceptance
> tests. To do this:
> 
>    - rename system1 to system-ubuntu-main
>    - rename system2 to system-fedora-misc
>    - split into build/check/acceptance
>    - remove -j from check stages
>    - use artifacts to save build stage
>    - add post acceptance template and use
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Message-Id: <20200622143204.12921-16-alex.bennee@linaro.org>
> 
> ---
> v2
>    - updated with danp's docker changes
>    - use needs instead of dependancies
>    - touch all the build files to prevent rebuild
> ---
>   .gitlab-ci.yml | 66 +++++++++++++++++++++++++++++++++++++++++++++++---
>   .travis.yml    | 23 ------------------
>   2 files changed, 63 insertions(+), 26 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index a7abc55a5c6..5ae8130bd1a 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,8 +1,12 @@
> +# Currently we have two build stages after our containers are built:
> +#  - build (for traditional build and test or first stage build)
> +#  - test (for test stages, using build artefacts from a build stage)
>   stages:
>     - containers
>     - containers-layer2
>     - containers-layer3
>     - build
> +  - test
>   
>   include:
>     - local: '/.gitlab-ci.d/edk2.yml'
> @@ -24,26 +28,82 @@ include:
>           ../configure --enable-werror $CONFIGURE_ARGS ;
>         fi
>       - make -j"$JOBS"
> -    - make -j"$JOBS" $MAKE_CHECK_ARGS
> +    - if test -n "$MAKE_CHECK_ARGS";
> +      then
> +        make $MAKE_CHECK_ARGS ;
> +      fi
> +
> +.native_test_job_template: &native_test_job_definition
> +  stage: test
> +  image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
> +  script:
> +    - cd build
> +    - find . -type f -exec touch {} +
> +    - make $MAKE_CHECK_ARGS
> +
> +.post_acceptance_template: &post_acceptance
> +  after_script:
> +    - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
> +    - du -chs $HOME/avocado/data/cache
>   
> -build-system1:
> +build:system-ubuntu-main:
>     <<: *native_build_job_definition
>     variables:
>       IMAGE: ubuntu2004
>       TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu lm32-softmmu
>         moxie-softmmu microblazeel-softmmu mips64el-softmmu m68k-softmmu ppc-softmmu
>         riscv64-softmmu sparc-softmmu
> +  artifacts:
> +    paths:
> +      - build
> +
> +check:system-ubuntu-main:
> +  <<: *native_test_job_definition
> +  needs:
> +    - job: build:system-ubuntu-main
> +      artifacts: true
> +  variables:
> +    IMAGE: ubuntu2004
>       MAKE_CHECK_ARGS: check
>   
> -build-system2:
> +acceptance:system-ubuntu-main:
> +  <<: *native_test_job_definition
> +  needs:
> +    - job: build:system-ubuntu-main
> +      artifacts: true
> +  variables:
> +    IMAGE: ubuntu2004
> +    MAKE_CHECK_ARGS: check-acceptance
> +
> +build:system-fedora-alt:
>     <<: *native_build_job_definition
>     variables:
>       IMAGE: fedora
>       TARGETS: tricore-softmmu unicore32-softmmu microblaze-softmmu mips-softmmu
>         riscv32-softmmu s390x-softmmu sh4-softmmu sparc64-softmmu x86_64-softmmu
>         xtensa-softmmu nios2-softmmu or1k-softmmu
> +  artifacts:
> +    paths:
> +      - build
> +
> +check:system-fedora-alt:
> +  <<: *native_test_job_definition
> +  needs:
> +    - job: build:system-fedora-alt
> +      artifacts: true
> +  variables:
> +    IMAGE: fedora
>       MAKE_CHECK_ARGS: check
>   
> +check:system-fedora-alt:

Copy-n-paste bug: This should be "acceptance:..." instead. By the way, I 
think the use of a colon after "check" or "acceptance" is a little bit 
confusing here, since this is also a character used by the YML syntax.
Could you name the jobs with a dash here instead? (check-system-fedora, 
acceptance-system-fedora, ...)

> +  <<: *native_test_job_definition
> +  needs:
> +    - job: build:system-fedora-alt
> +      artifacts: true
> +  variables:
> +    IMAGE: fedora
> +    MAKE_CHECK_ARGS: check-acceptance

  Thomas