Re: [RFC PATCH 0/1] ci: Speed up container stage

[Fabiano Rosas <farosas@suse.de>]

Hi Alex,

> Daniel P. Berrangé <berrange@redhat.com> writes:
>
>> On Thu, Feb 23, 2023 at 11:21:53AM -0300, Fabiano Rosas wrote:
>>> I'm not sure if this was discussed previously, but I noticed we're not
>>> pulling the images we push to the registry at every pipeline run.
>>> 
>>> I would expect we don't actually need to rebuild container images at
>>> _every_ pipeline run, so I propose we add a "docker pull" to the
>>> container templates. We already have that for the docker-edk2|opensbi
>>> images.
>>> 
>>> Some containers can take a long time to build (14 mins) and pulling
>>> the image first without building can cut the time to about 3
>>> mins. With this we can save almost 2h of cumulative CI time per
>>> pipeline run:
>>
>> The docker.py script that we're invoking is already pulling the
>> image itself eg to pick a random recent job:
>>
>>   https://gitlab.com/qemu-project/qemu/-/jobs/3806090058
>>
>> We can see
>>
>>   $ ./tests/docker/docker.py --engine docker build -t "qemu/$NAME" -f
>> "tests/docker/dockerfiles/$NAME.docker" -r
>> $CI_REGISTRY/qemu-project/qemu 03:54
>>   Using default tag: latest
>>   latest: Pulling from qemu-project/qemu/qemu/debian-arm64-cross
>>   bb263680fed1: Pulling fs layer
>>   ...snip...
>>
>> none the less it still went ahead and rebuilt the image from scratch
>> so something is going wrong here. I don't know why your change adding
>> an extra 'docker pull' would have any effect, given we're already
>> pulling, so I wonder if that's just coincidental apparent change
>> due to the initial state of your fork's container registery.
>>
>> Whenever I look at this I end up wishing out docker.py didn't exist
>> and that we could just directly do
>>
>>   - docker pull "$TAG"
>>   - docker build --cache-from "$TAG" --tag "$TAG" -f "tests/docker/$NAME.docker"
>>
>> as that sould be sufficient to build the image with caching.
>
> I think we should be ready to do that now as we have flattened all our
> dockerfiles. The only other thing that docker.py does is nicely add a
> final step for the current user so you can ensure all files generated in
> docker cross compile images are still readable on the host.
>

Just so you know this command line worked:

docker build --cache-from $TAG --tag $TAG --build-arg BUILDKIT_INLINE_CACHE=1 \
  -f "tests/docker/dockerfiles/$NAME.docker" "."

building the cache: https://gitlab.com/farosas/qemu/-/jobs/3825838177
using the cache:    https://gitlab.com/farosas/qemu/-/jobs/3825926944

But we might still have this issue:

commit 6ddc3dc7a882f2e7200fa7fecf505a8d0d8bbea9
Author: Daniel P. Berrangé <berrange@redhat.com>
Date:   Fri Jul 9 15:29:35 2021 +0100

    tests/docker: don't use BUILDKIT in GitLab either
    
    Using BUILDKIT breaks with certain container registries such as CentOS,
    with docker build reporting an error such as
    
      failed to solve with frontend dockerfile.v0:
      failed to build LLB: failed to load cache key:
      unexpected status code
      https://registry.centos.org/v2/centos/manifests/7:
      403 Forbidden

We might need to go the route of skipping the docker build when the
docker pull succeeds.