Re: [PATCH RFC 2/3] gitlab: build all container images during CI

[Thomas Huth <thuth@redhat.com>]

On 25/06/2020 12.24, Daniel P. Berrangé wrote:
> On Thu, Jun 25, 2020 at 12:14:33PM +0200, Thomas Huth wrote:
>> On 22/06/2020 17.33, Daniel P. Berrangé wrote:
>>> We have a number of container images in tests/docker/dockerfiles
>>> that are intended to provide well defined environments for doing
>>> test builds. We want our CI system to use these containers too.
>>>
>>> This introduces builds of all of them as the first stage in the
>>> CI, so that the built containers are available for later build
>>> jobs. The containers are setup to use the GitLab container
>>> registry as the cache, so we only pay the penalty of the full
>>> build when the dockerfiles change. The main qemu-project/qemu
>>> repo is used as a second cache, so that users forking QEMU will
>>> see a fast turnaround time on their CI jobs.
>>>
>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>> ---
>>>    .gitlab-ci.d/containers.yml | 248 ++++++++++++++++++++++++++++++++++++
>>>    .gitlab-ci.yml              |   3 +
>>>    2 files changed, 251 insertions(+)
>>>    create mode 100644 .gitlab-ci.d/containers.yml
>>>
>>> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
>>> new file mode 100644
>>> index 0000000000..ea1edbb196
>>> --- /dev/null
>>> +++ b/.gitlab-ci.d/containers.yml
>>> @@ -0,0 +1,248 @@
>>> +
>>> +
>>> +.container_job_template: &container_job_definition
>>> +  image: docker:stable
>>> +  stage: containers
>>> +  services:
>>> +    - docker:dind
>>> +  before_script:
>>> +    - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest"
>>> +    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/ci-$NAME:latest"
>>> +    - docker info
>>> +    - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
>>> +  script:
>>> +    - docker pull "$TAG" || docker pull "$COMMON_TAG" || true
>>> +    - sed -i -e "s,FROM qemu:,FROM $CI_REGISTRY_IMAGE/ci-," tests/docker/dockerfiles/$NAME.docker
>>> +    - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "tests/docker/dockerfiles/$NAME.docker" tests/docker/dockerfiles
>>> +    - docker push "$TAG"
>>> +  after_script:
>>> +    - docker logout
>>
>> .gitlab-ci.d/edk2.yml uses a "changes" rule to only run the pipeline if
>> something really has been changed. Could you use something similar here?
>> E.g.:
>>
>> rules:
>>   - changes:
>>     - .gitlab-ci.d/containers.yml
>>     - tests/docker/*
>>     - tests/docker/dockerfiles/*
>>
>> ?
> 
> If the OS distro base image changes, we'll never pick it up with that
> kind of filtering.  For the main gitlab.com/qemu-project/qemu  you
> could configure a nightly/weekly/whatever job to force rebuild on a
> periodic basis to pick up base image changes.  The downside of this
> is that any users who fork qemu won't have that periodic job and so
> will be testing their work against potentially outdated content.
> 
> Having said all that, I'm not 100% convinced I'm actually picking
> up changed base images right now anyway, given our use of caching.
> 
> It is possible that I would need todo an explict "docker pull" of
> the base image to force it to trigger a refresh othrewise I have
> a feeling we're always cached.

But currently, each of the container stages currently takes > 2 minutes, 
even with the cached containers. I had a quick look, and it takes 7 
minutes 'till the "build" stage begins. So all the advantages of not 
having to do "yum/apt-get install" in the build containers anymore seem 
to be crushed by the time that the three container stages take now?

  Thomas