Re: [PATCH RFC 2/3] gitlab: build all container images during CI

["Daniel P. Berrangé" <berrange@redhat.com>]

On Mon, Jun 22, 2020 at 04:33:17PM +0100, Daniel P. Berrangé wrote:
> We have a number of container images in tests/docker/dockerfiles
> that are intended to provide well defined environments for doing
> test builds. We want our CI system to use these containers too.
> 
> This introduces builds of all of them as the first stage in the
> CI, so that the built containers are available for later build
> jobs. The containers are setup to use the GitLab container
> registry as the cache, so we only pay the penalty of the full
> build when the dockerfiles change. The main qemu-project/qemu
> repo is used as a second cache, so that users forking QEMU will
> see a fast turnaround time on their CI jobs.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>  .gitlab-ci.yml              |   3 +
>  2 files changed, 251 insertions(+)
>  create mode 100644 .gitlab-ci.d/containers.yml

Thinking about this at 3am with insomnia, I started considering possible
ways this could result in CI failures. I came up with

 - Distro repos are unavialable due to transient network/mirror problems
 
 - Distro pushes bad packages or changes something that invalidates QEMU
   assumptions

The first one can hit at any time, but I think that we're reasonably well
insulated from it due to our usage of caching. Only a small number of CI
jobs are going to actually trigger a full rebuild of an image, most will
just hit the cache.

The second one is probably not likely, *provided* we only use stable branches
of distros. It looks like we're ok in that regard as we're not using Debian
Sid, or Fedora rawhide for any images currently.

If we did want to reduce the risk though, we could mark some of the
container jobs as non-fatal. We could mark the subset of images that are
not actually required by later build jobs that we currently have.

> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> new file mode 100644
> index 0000000000..ea1edbb196
> --- /dev/null
> +++ b/.gitlab-ci.d/containers.yml
> @@ -0,0 +1,248 @@
> +
> +
> +.container_job_template: &container_job_definition
> +  image: docker:stable
> +  stage: containers
> +  services:
> +    - docker:dind
> +  before_script:
> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
> +    - docker info
> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> +  script:
> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
> +    - docker push "$TAG"
> +  after_script:
> +    - docker logout
> +
> +amd64-centos7-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: centos7
> +
> +amd64-centos8-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: centos8
> +
> +amd64-debian10-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian10
> +
> +amd64-debian11-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian11
> +
> +amd64-debian9-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian9
> +
> +amd64-debian9-mxe-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian9-container']
> +  variables:
> +    NAME: debian9-mxe
> +
> +alpha-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-alpha-cross
> +
> +amd64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-amd64-cross
> +
> +amd64-debian-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-amd64
> +
> +arm64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-arm64-cross
> +
> +arm64-test-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian11-container']
> +  variables:
> +    NAME: debian-arm64-test-cross
> +
> +armel-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-armel-cross
> +
> +armhf-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-armhf-cross
> +
> +hppa-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-hppa-cross
> +
> +m68k-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-m68k-cross
> +
> +mips64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips64-cross
> +
> +mips64el-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips64el-cross
> +
> +mips-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mips-cross
> +
> +mipsel-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-mipsel-cross
> +
> +powerpc-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-powerpc-cross
> +
> +ppc64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-ppc64-cross
> +
> +ppc64el-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-ppc64el-cross
> +
> +riscv64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-riscv64-cross
> +
> +s390x-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-s390x-cross
> +
> +sh4-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-sh4-cross
> +
> +sparc64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian10-container']
> +  variables:
> +    NAME: debian-sparc64-cross
> +
> +tricore-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer2
> +  needs: ['amd64-debian9-container']
> +  variables:
> +    NAME: debian-tricore-cross
> +
> +win32-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer3
> +  needs: ['amd64-debian9-mxe-container']
> +  variables:
> +    NAME: debian-win32-cross
> +
> +win64-debian-cross-container:
> +  <<: *container_job_definition
> +  stage: containers-layer3
> +  needs: ['amd64-debian9-mxe-container']
> +  variables:
> +    NAME: debian-win64-cross
> +
> +xtensa-debian-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: debian-xtensa-cross
> +
> +cris-fedora-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora-cris-cross
> +
> +amd64-fedora-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora
> +
> +i386-fedora-cross-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: fedora-i386-cross
> +
> +amd64-ubuntu1804-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu1804
> +
> +amd64-ubuntu2004-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu2004
> +
> +amd64-ubuntu-container:
> +  <<: *container_job_definition
> +  variables:
> +    NAME: ubuntu
> +
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 9fdc752ea6..72d688875f 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,10 +1,13 @@
>  stages:
>    - containers
> +  - containers-layer2
> +  - containers-layer3
>    - build
>  
>  include:
>    - local: '/.gitlab-ci.d/edk2.yml'
>    - local: '/.gitlab-ci.d/opensbi.yml'
> +  - local: '/.gitlab-ci.d/containers.yml'
>  
>  .update_apt_template: &before_script_apt
>   before_script:
> -- 
> 2.24.1
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|