Re: [PATCH RFC 2/3] gitlab: build all container images during CI

["Alex Bennée" <alex.bennee@linaro.org>]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Mon, Jun 22, 2020 at 04:33:17PM +0100, Daniel P. Berrangé wrote:
>> We have a number of container images in tests/docker/dockerfiles
>> that are intended to provide well defined environments for doing
>> test builds. We want our CI system to use these containers too.
>> 
>> This introduces builds of all of them as the first stage in the
>> CI, so that the built containers are available for later build
>> jobs. The containers are setup to use the GitLab container
>> registry as the cache, so we only pay the penalty of the full
>> build when the dockerfiles change. The main qemu-project/qemu
>> repo is used as a second cache, so that users forking QEMU will
>> see a fast turnaround time on their CI jobs.
>> 
>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>> ---
>>  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>>  .gitlab-ci.yml              |   3 +
>>  2 files changed, 251 insertions(+)
>>  create mode 100644 .gitlab-ci.d/containers.yml
>
> Thinking about this at 3am with insomnia, I started considering possible
> ways this could result in CI failures. I came up with
>
>  - Distro repos are unavialable due to transient network/mirror problems
>  
>  - Distro pushes bad packages or changes something that invalidates QEMU
>    assumptions
>
> The first one can hit at any time, but I think that we're reasonably well
> insulated from it due to our usage of caching. Only a small number of CI
> jobs are going to actually trigger a full rebuild of an image, most will
> just hit the cache.
>
> The second one is probably not likely, *provided* we only use stable branches
> of distros. It looks like we're ok in that regard as we're not using Debian
> Sid, or Fedora rawhide for any images currently.

We use debian11 (which I think is debian-testing) for one of the aarch64
compilers because we need fairlyu bleeding edge gcc's but we won't go
back to sid anytime soon.

>
> If we did want to reduce the risk though, we could mark some of the
> container jobs as non-fatal. We could mark the subset of images that are
> not actually required by later build jobs that we currently have.
>
>> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
>> new file mode 100644
>> index 0000000000..ea1edbb196
>> --- /dev/null
>> +++ b/.gitlab-ci.d/containers.yml
>> @@ -0,0 +1,248 @@
>> +
>> +
>> +.container_job_template: &container_job_definition
>> +  image: docker:stable
>> +  stage: containers
>> +  services:
>> +    - docker:dind
>> +  before_script:
>> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
>> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
>> +    - docker info
>> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
>> +  script:
>> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
>> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
>> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
>> +    - docker push "$TAG"
>> +  after_script:
>> +    - docker logout
>> +
>> +amd64-centos7-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: centos7
>> +
>> +amd64-centos8-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: centos8
>> +
>> +amd64-debian10-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: debian10
>> +
>> +amd64-debian11-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: debian11
>> +
>> +amd64-debian9-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: debian9
>> +
>> +amd64-debian9-mxe-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian9-container']
>> +  variables:
>> +    NAME: debian9-mxe
>> +
>> +alpha-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-alpha-cross
>> +
>> +amd64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-amd64-cross
>> +
>> +amd64-debian-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-amd64
>> +
>> +arm64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-arm64-cross
>> +
>> +arm64-test-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian11-container']
>> +  variables:
>> +    NAME: debian-arm64-test-cross
>> +
>> +armel-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-armel-cross
>> +
>> +armhf-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-armhf-cross
>> +
>> +hppa-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-hppa-cross
>> +
>> +m68k-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-m68k-cross
>> +
>> +mips64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-mips64-cross
>> +
>> +mips64el-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-mips64el-cross
>> +
>> +mips-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-mips-cross
>> +
>> +mipsel-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-mipsel-cross
>> +
>> +powerpc-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-powerpc-cross
>> +
>> +ppc64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-ppc64-cross
>> +
>> +ppc64el-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-ppc64el-cross
>> +
>> +riscv64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-riscv64-cross
>> +
>> +s390x-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-s390x-cross
>> +
>> +sh4-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-sh4-cross
>> +
>> +sparc64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian10-container']
>> +  variables:
>> +    NAME: debian-sparc64-cross
>> +
>> +tricore-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer2
>> +  needs: ['amd64-debian9-container']
>> +  variables:
>> +    NAME: debian-tricore-cross
>> +
>> +win32-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer3
>> +  needs: ['amd64-debian9-mxe-container']
>> +  variables:
>> +    NAME: debian-win32-cross
>> +
>> +win64-debian-cross-container:
>> +  <<: *container_job_definition
>> +  stage: containers-layer3
>> +  needs: ['amd64-debian9-mxe-container']
>> +  variables:
>> +    NAME: debian-win64-cross
>> +
>> +xtensa-debian-cross-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: debian-xtensa-cross
>> +
>> +cris-fedora-cross-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: fedora-cris-cross
>> +
>> +amd64-fedora-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: fedora
>> +
>> +i386-fedora-cross-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: fedora-i386-cross
>> +
>> +amd64-ubuntu1804-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: ubuntu1804
>> +
>> +amd64-ubuntu2004-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: ubuntu2004
>> +
>> +amd64-ubuntu-container:
>> +  <<: *container_job_definition
>> +  variables:
>> +    NAME: ubuntu
>> +
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 9fdc752ea6..72d688875f 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -1,10 +1,13 @@
>>  stages:
>>    - containers
>> +  - containers-layer2
>> +  - containers-layer3
>>    - build
>>  
>>  include:
>>    - local: '/.gitlab-ci.d/edk2.yml'
>>    - local: '/.gitlab-ci.d/opensbi.yml'
>> +  - local: '/.gitlab-ci.d/containers.yml'
>>  
>>  .update_apt_template: &before_script_apt
>>   before_script:
>> -- 
>> 2.24.1
>> 
>
> Regards,
> Daniel


-- 
Alex Bennée