Re: [PATCH 11/11] tests/gitlab: use kaniko to build images

["Daniel P. Berrangé" <berrange@redhat.com>]

On Thu, Mar 30, 2023 at 11:17:41AM +0100, Daniel P. Berrangé wrote:
> On Thu, Mar 30, 2023 at 11:11:41AM +0100, Alex Bennée wrote:
> > Apparently the docker-in-docker approach has some flaws including
> > needing privileged mode to run and being quite slow. An alternative
> > approach is to use Google's kaniko tool. It also works across
> > different gitlab executors.
> > 
> > Following the gitlab example code we drop all the direct docker calls
> > and usage of the script and make a direct call to kaniko and hope the
> > images are cacheable by others.
> > 
> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > Message-Id: <20230224180857.1050220-8-alex.bennee@linaro.org>
> > 
> > ---
> > v2
> >   - add danpb's --cache suggestions
> > ---
> >  .gitlab-ci.d/container-template.yml | 22 ++++++++++------------
> >  1 file changed, 10 insertions(+), 12 deletions(-)
> > 
> > diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> > index 519b8a9482..cd8e0a1ff6 100644
> > --- a/.gitlab-ci.d/container-template.yml
> > +++ b/.gitlab-ci.d/container-template.yml
> > @@ -1,21 +1,19 @@
> >  .container_job_template:
> >    extends: .base_job_template
> > -  image: docker:stable
> > +  image:
> > +    name: gcr.io/kaniko-project/executor:v1.9.0-debug
> > +    entrypoint: [""]
> >    stage: containers
> > -  services:
> > -    - docker:dind
> >    before_script:
> >      - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
> >      - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
> > -    - apk add python3
> > -    - docker info
> > -    - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD"
> >    script:
> >      - echo "TAG:$TAG"
> >      - echo "COMMON_TAG:$COMMON_TAG"
> > -    - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
> > -      --build-arg BUILDKIT_INLINE_CACHE=1
> > -      -f "tests/docker/dockerfiles/$NAME.docker" "."
> > -    - docker push "$TAG"
> > -  after_script:
> > -    - docker logout
> > +    - /kaniko/executor
> > +          --reproducible
> > +          --context "${CI_PROJECT_DIR}"
> > +          --cache=true
> > +          --cache-repo "${COMMON_TAG}"
> 
> IIRC with docker if we told it to cache we would have to first have done
> a  'docker pull $COMMON_TAG' as it wouldn't pull down the image if
> it was not already local. I'm fuzzy on whether kaniko has the same
> need or not ?  I guess we were broken already in that respect as
> we already uses --cache-from with docker without a docker pull

Oh never mind, because we're not docker-in-docker, we can't pull the
image tag down locally, and as discussed on IRC, caching works in a
very different way. kaniko wants to be able to push & pull in the
cache-repo itself.

I'm inclined to think we're better off ignoring layer caching and instead
focus on entirely skipping execution of kaniko if we know the dockerfile
has not changed eg something along the lines of:

   manifest=$(curl ....some registry URL to fetch image metadata)
   oldchecksum=$(...extract a LABEL from metadata container dockerfile sha256)
   newchecksum=$(sha256sum tests/docker/dockerfiles/$NAME.docker)

   if test $oldchecksum != $newchecksum -o -n $QEMU_FORCE_REBUILD"
   then
      - /kaniko/executor
            --reproducible
            --context "${CI_PROJECT_DIR}"
            --dockerfile "${CI_PROJECT_DIR}/tests/docker/dockerfiles/$NAME.docker"
	    --label DKR_CHECKSUM=$newchecksum
            --destination "${TAG}"
   fi


And then have a weekly pipeline on sundays that sets QEMU_FORCE_REBUILD=1
so that we pick up changes from the distro base images, and/or package
repes regularly.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|