Re: [RFC PATCH 0/1] ci: Speed up container stage

["Daniel P. Berrangé" <berrange@redhat.com>]

On Thu, Feb 23, 2023 at 05:30:58PM -0300, Fabiano Rosas wrote:
> 
> Hi Alex,
> 
> > Daniel P. Berrangé <berrange@redhat.com> writes:
> >
> >> On Thu, Feb 23, 2023 at 11:21:53AM -0300, Fabiano Rosas wrote:
> >>> I'm not sure if this was discussed previously, but I noticed we're not
> >>> pulling the images we push to the registry at every pipeline run.
> >>> 
> >>> I would expect we don't actually need to rebuild container images at
> >>> _every_ pipeline run, so I propose we add a "docker pull" to the
> >>> container templates. We already have that for the docker-edk2|opensbi
> >>> images.
> >>> 
> >>> Some containers can take a long time to build (14 mins) and pulling
> >>> the image first without building can cut the time to about 3
> >>> mins. With this we can save almost 2h of cumulative CI time per
> >>> pipeline run:
> >>
> >> The docker.py script that we're invoking is already pulling the
> >> image itself eg to pick a random recent job:
> >>
> >>   https://gitlab.com/qemu-project/qemu/-/jobs/3806090058
> >>
> >> We can see
> >>
> >>   $ ./tests/docker/docker.py --engine docker build -t "qemu/$NAME" -f
> >> "tests/docker/dockerfiles/$NAME.docker" -r
> >> $CI_REGISTRY/qemu-project/qemu 03:54
> >>   Using default tag: latest
> >>   latest: Pulling from qemu-project/qemu/qemu/debian-arm64-cross
> >>   bb263680fed1: Pulling fs layer
> >>   ...snip...
> >>
> >> none the less it still went ahead and rebuilt the image from scratch
> >> so something is going wrong here. I don't know why your change adding
> >> an extra 'docker pull' would have any effect, given we're already
> >> pulling, so I wonder if that's just coincidental apparent change
> >> due to the initial state of your fork's container registery.
> >>
> >> Whenever I look at this I end up wishing out docker.py didn't exist
> >> and that we could just directly do
> >>
> >>   - docker pull "$TAG"
> >>   - docker build --cache-from "$TAG" --tag "$TAG" -f "tests/docker/$NAME.docker"
> >>
> >> as that sould be sufficient to build the image with caching.
> >
> > I think we should be ready to do that now as we have flattened all our
> > dockerfiles. The only other thing that docker.py does is nicely add a
> > final step for the current user so you can ensure all files generated in
> > docker cross compile images are still readable on the host.
> >
> 
> Just so you know this command line worked:
> 
> docker build --cache-from $TAG --tag $TAG --build-arg BUILDKIT_INLINE_CACHE=1 \
>   -f "tests/docker/dockerfiles/$NAME.docker" "."
> 
> building the cache: https://gitlab.com/farosas/qemu/-/jobs/3825838177
> using the cache:    https://gitlab.com/farosas/qemu/-/jobs/3825926944
> 
> But we might still have this issue:
> 
> commit 6ddc3dc7a882f2e7200fa7fecf505a8d0d8bbea9
> Author: Daniel P. Berrangé <berrange@redhat.com>
> Date:   Fri Jul 9 15:29:35 2021 +0100
> 
>     tests/docker: don't use BUILDKIT in GitLab either
>     
>     Using BUILDKIT breaks with certain container registries such as CentOS,
>     with docker build reporting an error such as
>     
>       failed to solve with frontend dockerfile.v0:
>       failed to build LLB: failed to load cache key:
>       unexpected status code
>       https://registry.centos.org/v2/centos/manifests/7:
>       403 Forbidden
> 
> We might need to go the route of skipping the docker build when the
> docker pull succeeds.

I don't recall what made QEMU special in that respect, but we never
set this variable in any libvirt project and didn't see any obvious
problems. So I'm inclined to say it was something the docker.py
does that's caused the problem

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|